There were two things I changed after which the issue disappeared.
a) Found that an app pool was shared by two web app. Created a new App pool and assigned individual app pool to both web apps. Although one of the web app was not being used.
b) Changed the diagnostic logging for "All" events to Error only to remove excessive log writing.
After above 2 changes the following happened:
a) Error from indexing on Large list (50,000) items disappeared. Were getting same error earlier i.e. Error in data web service...
b) The crawl time reduced from 18 hours to 2 hours.
Long term solution: Migrate to 64 bit environment.