I'd say forget about cleric, drop the wisdom (by 4), cha (by 1) and str (by 2) and put those points in DEX (1) and INT (2). Then go for Wizard instead to get spells.
Advantages: better damage (swashbukler int bonus counts fully for both off- and main hand, strength bonus only counts for half on your off-hand), more skillpoints, more class skills, better skill levels of the skills you're most likely to use, ability to cast spells like Ghostly Visage and Invisibility, higher initial AC and AB (and higher max total dex including equipment, of 26 at level 20). Disadvantages: lower ability/skill checks based on str and wis, lower will saves.
Think it's worth the trade-off.
Advantages: better damage (swashbukler int bonus counts fully for both off- and main hand, strength bonus only counts for half on your off-hand), more skillpoints, more class skills, better skill levels of the skills you're most likely to use, ability to cast spells like Ghostly Visage and Invisibility, higher initial AC and AB (and higher max total dex including equipment, of 26 at level 20). Disadvantages: lower ability/skill checks based on str and wis, lower will saves.
Think it's worth the trade-off.
Comment