I go to a lot of open mics. I've been to one or two (or three) a week for the last year and a half andI can tell you what you need to know so that you don't end up being 'that guy/girl who sucked at the open mic.'
Don't Play Long Songs - This is possibly the most fundamental mistake. Open mic is not the place for your Magnum Opus. The crowd has already heard a lot of soul-searching acoustic guitar ballads at this point and someone with a very long song only irritates people.
Shut up. Shut up. Shut up! Just be quiet if you feel like talking and start playing something right away. The audience is made up of other musicians who are only there because it's polite to stay for the other artists; they don't care about you. Limit your words to 'my name is____, my website is www.______ and my next show is at ____'. Nothing else. We don't care. When you ramble on you look very unprofessional and people assume that you are not a serious musician. This can cripple your performance before you strum a note. Shut up and play, then leave.
Don't Play Long Empty Chord Progressions. This happens too often. Some guy sings a verse, then strums the chords with no vocals for a minute. I'm sure he's thinking, 'a guitar solo could go here' or something, but it's extremely boring for us. You need to have your songs arranged so that the crowd is not just waiting for you to stop wasting their time with your pointless G- C - D drudgery. Even a young music student can learn to shorten boring sections of a song. (It is OK to play chord progessions that sound good, we are only complaining about chord progessions that are boring without something to go with it.)
Don't Scream/Yell. Sounds obvious, but too many people (young and old, actually) don't get it. You're not performing a stadium show. When you shout 'Yeeeehaw!' or yell your vocals you are doing so in a small space with a sound system that doesn't handle those types of sounds. You will be horrified to see that people listening will actually cover their ears and sometimes leave. Sure, lots of good music involves screaming. Open mic is the wrong place for it; you need to tone it down a bit.
Know You Limits and Keep it Simple - There is a guy I know who plays a nice, simple song at open mics in my area. It's an easy song to play and sing, but everyone remembers it and wants him to play it again and again. This guy knows his limits and keeps things simple. I have seen others try to sing their highest notes and play songs that they can't handle and they look foolish doing so. They might be better musicians than the friend I mentioned, but his performance always stands out. Last night, a young lady was playing guitar and singing with a friend. She is not very experienced on the guitar and made a few noticeable mistakes, but the songs were good and the musicians kept playing and smiling through the mistakes. It was a better set than most of the other groups that played even though her guitar playing was weaker. They arranged the set to accommodate Their skill, in doing so, a weaker musician carried a very strong performance that the more experienced musicians in the room didn't usually do.
Don't Leave Until Everyone Has Played and Buy Some Coffee - This is critical. When you are finished, stick around. It's not fair for you to leave so that the last people to sign up have no one to play for. It's rude and it won't get you any fans. In fact, those present will tell people not to go to your shows. Also, if you are at a coffee shop (where most of these things are held) you should make a habit of buying some coffee. Some venues do not allow certain acts to play because at open mics those particular acts don't buy anything and don't stick around for the other artists. (There are some expections, here. Sometimes you have to be at work early and can't stay all night, it happens. Also, at one location, the person who serves drinks was rude to me so I'm not buying anything there ever again.)
In Conclusion - Don't ruin your musical career buy being that jerk who screams long songs at the crowd, then leaves right away and never buys anything. By impressing people at an open mic and being friendly, you will encourage people to attend your shows. It's slow, but you can build fans this way who tell their friends to check you out at a show, and that's what you're trying to do.