Les Healey had been taken off the woman's case but arranged to meet her in secret, the General Social Care Council heard.
Her face was twitching and there were sores on her back but Healey drove away and did not inform colleagues.
The secret visit came to light following claims he had groped the married woman's breasts and bottom at her home near Whitby.
Healey, who lived in Whitby, was assigned her case in 2005 after Scarborough Social Services had concerns about her children.
He saw her on 12 January 2006 at her home and allegedly assaulted her there.
Tracey Whitfield, a service manager for North Yorkshire County Council, interviewed the woman known as Ms A after she came forward in October that year.
"She (Ms A] said as he was leaving, he pushed her up against the wall.
"He groped her. He felt her waist and breasts, and felt her bottom over her trousers."
She said Mr Healey became aroused.
Ms Whitfield said that three days later Healey phoned the woman up.
"He asked her if she would like to go for a coffee," she said.
"When she said 'no', he said, 'You won't tell anyone about this, will you? I'll lose my job if you do'. Ms A said she wouldn't."
Ms Whitfield asked Ms A why she had not reported the matter for nine months.
"She didn't want the discussion because she was concerned he may take her eldest son away."
Ms Whitfield said she interviewed Healey on 12 October that year and he claimed he was the victim.
"He told me they were sitting next to each other on the sofa," said Ms Whitfield.
"She rested his hands on her arms, he said.
"I moved to the kitchen but she came forward and had her hands around his waist. She wanted to thank him."
Ms Whitfield said Healey told her to stop several times and tried to "make light of the situation".
"He said, 'What would your husband think if he saw us like this?'"
During the interview a colleague asked if Healey had visited the woman again.
Ms Whitfield said she was "taken aback" to hear about an incident on April 20 that year.
The woman known as Ms A had been transferred to another social worker following Healey's report of sexual harassment and he should have had nothing more to do with her, she said.
But he told her he arranged the off-the-record call on his way home to Whitby, it was said.
Healey told her that when he got into her home Ms A was on her hands and knees, looking at a picture of her children.
"She had an open sore across her lower back," said Ms Whitfield.
"She said, 'Look at my lovely children'. She couldn't put sentences together. The side of her face was twitching and she told him her husband had left her."
Ms Whitfield said that according to Healey he went upstairs to "think". And a few minutes went back to her.
"He told her to sort herself out or she could die. He then drove away."
Ms Whitfield said Ms A was clearly in a "very serious state" and it was Healey's duty to report the matter, and possibly call an ambulance.
"But he didn't do any of that," she said.
Healey, who no longer lives in the UK, is not attending today's hearing but and denies all the allegations.
He resigned in 2008 after his bosses at North Yorkshire County Council found out about a secret affair with another woman under his protection.
The social worker is charged with contacting Ms A in 2006 when he pushed her against a wall, attempting to kiss her, and touching her breasts and bottom. Following the visit he is accused of swearing her to secrecy.
It later came out that Healey asked his line manager not to make a complaint following Ms A's alleged sexual assault, the hearing heard
Lucy Monk said that the day after the incident Healey came into her office shaking on January 10.
"His first words to me were I've been sexually assaulted and his hands were shaking," she said.
But he told Ms Monk, who worked at Scarborough Social Services, that Ms A would feel "rejected" if a letter was sent to her, it was said.
Healey also told her he had faced allegations of sexual assault himself, in 2001, and was concerned lest they surface again.
She said: "At time I thought he was embarrassed and upset. He was absolutely adamant he didn't want anyone to know (about the assault]. And didn't want it to go any further."
Given his opposition Ms Monk said she decided to write only the words 'an incident' on the supervision record.
He faces separate allegations in relation to two other clients.
In 1997 he allegedly told Ms B user he wanted sex with her. Then, after abusing his position, got hold of her address and sent her a Christmas card.
In 2008 he is charged with breaching professional boundaries by forming a sexual relationship with a Ms C; making her a loan of between £4,000 and £5,000; acting as her child's godparent; taking her and her family on holiday; and repeatedly contacting her despite instructions to the contrary.
The hearing continues.