The German railways site gives all train connections including slow milk-run trains, but you have to give it the right instructions.

After entering your dates, destination, etc., be sure to click on "only local transport" under "means of transport" at this page:

http://reiseauskunft.bahn.de/bin/query.exe/en

When the itineraries pop up, click the white-on-red horizontal arrow at the left of each itinerary, and the changes of train will appear.

Then you can also find every stop for every train mentioned too. Where the changes are listed, move your cursor over to the "Products" column and click on the train numbers - you'll get a pop-up with every station that train passes through.

The above will help you find towns on the most direct routes. If you want to zig and zag, you'll need to enter "stopover" points (near the top, under "destination.)

Utrecht, Maastricht and Aachen spring to mind as possible stopovers between Amsterdam and Cologne.

I think I would prefer the route from Paris.. Reims, Metz and or Luxembourg, Trier, down the Mosel valley with a dozen "quaint" small towns to choose from on the river.

Metz, in France, has a handsome cathedral and a new branch of the Pompidou museum. It is a pleasant place for Franco/Belgium/German food and drink. Too bad the train station and museum are a couple of kilometres from the historic centre.
Maastricht, full of history plus a famous euro accord, and Aachan, centre of the Holy Roman Empire with some Charlemagne relics to see, are both worth a stopover. It depends on how you manage your rail connections.