"Now Joseph was handsome in form and appearance. And it came to pass after these things that his master's wife cast longing eyes on Joseph, and she said, 'Lie with me.' But he refused and said to his master's wife, 'Look, my master does not know what is with me in the house, and he has committed all that he has into my hand. There is no one greater in this house than I, nor has he kept back anything from me but you, because you are his wife. How then can I do this great wickedness, and sin against God?' So it was, as she spoke to Joseph day by day, that he did not heed her, to lie with her or to be with her" (Gen. 39:6-10).
Apparently, Joseph was well built and good looking, and Potiphar's wife had noticed. She lusted after him and made her desires known to him. She wanted to have sex with him and tried wearing him down, since he resisted her advances. It seems likely, based on the authority and privileges Joseph had been granted, that had he desired to sleep with her, he could have gotten away with such without Potiphar finding out. But Joseph was a man of integrity. Whether he found Potiphar's wife attractive or not was irrelevant--she belonged to another man! To become intimate with her would have been a sin against Potiphar and against God! Joseph has no interest in engaging in wicked behavior (that is what adultery is) or betraying Potiphar's trust. She, like Delilah, is persistent, but Joseph had strength that Samson did not possess (cf. Jud. 16). I am reminded here of I Corinthians 10:13 - "No temptation has overtaken you except such as is common to man; but God is faithful, who will not allow you to be tempted beyond what you are able, but with the temptation will also make the way of escape, that you may be able to bear it." In Joseph's situation, his only "escape" from temptation was endurance (and so it is with us sometimes). He could not leave his master's house, but he would not give in to his master's wife. He tried to avoid her as much as possible, which was certainly wise (cf. I Cor. 15:33).
"But it happened about this time, when Joseph went into the house to do his work, and none of the men of the house was inside, that she caught him by his garment, saying, 'Lie with me.' But he left his garment in her hand and fled outside, then she called to the men of her house and spoke to them, saying, 'See, he has brought in to us a Hebrew to mock us. He came in to me to lie with me, and I cried out with a loud voice. And it happened, when he heard that I lifted my voice and cried out, that he left his garment with me, and fled and went outside'" (Gen. 39:11-15).
Potiphar's wife looked for (or perhaps even arranged) an opportune time when she could be alone with Joseph. She must have believed that Joseph would have intercourse with her if they were alone and if she was aggressive physically. She was wrong. Although she had a firm hold of his garment, he abandoned it and fled outside to escape the seductress. When she was rejected this time, however, her passion for Joseph turned to hate (cf. II Sam. 13:15)! With his garment in hand, she resorted to lies and vengeance. She first tells the other servants her made up story and then prepares to tell Potiphar later that day.
"So she kept his garment with her until his master came home. Then she spoke to him with words like these, saying, 'The Hebrew servant whom you brought to us came in to me to mock me; so it happened, as I lifted my voice and cried out, that he left his garment with me and fled outside.' So it was, when his master heard the words which his wife spoke to him, saying, 'Your servant did to me after this manner,' that his anger was aroused. Then Joseph's master took him and put him into the prison, a place where the king's prisoners were confined. And he was there in the prison" (Gen. 39:16-20).
One might assume that Potiphar, after hearing his wife's story, was angry with Joseph, but I'm not convinced that such is the case. Had he believed that Joseph had actually tried to rape his wife, he (as the head of the executioners) likely would have had him executed! Instead, Potiphar puts Joseph in with Pharaoh's prisoners, which was a preferred place in prison. I believe Potiphar is angry with his wife because this situation will cost him his best servant. It was a "he said, she said" situation, essentially, and it would not have been appropriate to favor his servant over his wife, so Joseph has to go (though I'm not convinced Potiphar believes his wife here).