Let me hazard a wild guess: the system of passwords you use on the internet – for accessing online banking, email, shopping sites, Twitter and Facebook accounts – is a mess. You know perfectly well what you ought to be doing: for each site you visit, you should be choosing a different, complex sequence of letters, numbers and symbols, and then memorizing it. (That’s rule number one of the conventional wisdom on passwords: never, ever write them down.) But you don’t do this, because you weren’t blessed with a brain that’s capable of such feats. So instead you use the same familiar words for every site – your dog’s name, the name of your street – with occasional ingenious permutations, such as adding “123” at the end. Or maybe you do try to follow the rules, in which case you’re probably constantly getting locked out of your bank account or trying to remember the answers to various absurd security questions. (“What was your favourite sport as a child?” I’m now asked, though my real favourite sport was finding ways to dodge PE. One question at the iTunes Store asks users to nominate their “least favourite car”.) And things are getting worse: these days, you find yourself forced to choose passwords with both upper- and lower-case letters, and what normal human being can remember multiple combinations of those? Not you, that’s for sure.

One reason not to feel too guilty about your bad password behaviour is that it seems to be almost universal. Last month, an analysis of leaked pin numbers revealed that about one in 10 of us uses “1234”; a recent security breach at Yahoo showed that thousands of users’ passwords were either “password”, “welcome”, “123456” or “ninja”. People choose terrible passwords even when more is at stake than their savings: among military security specialists, it’s well-known that at the height of the cold war, the “secret unlocking code” for America’s nuclear missiles was 00000000. Five years ago, Newsnight revealed that, until 1997, some British nuclear missiles were armed by turning a key in what was essentially a bike lock. To choose whether the bomb should explode in the air or on the ground, you turned dials using an Allen key, Ikea-style. There weren’t any passcodes at all. Speed of retaliation, in the event of an enemy attack, counted for everything.

The parlous state of our passwords is the result of a different arms race, between malicious hackers and “white-hat” security testers. But when you talk to some of the people most deeply involved, it soon becomes clear that the conventional wisdom is flawed. For example: writing down your passwords may be an excellent plan. Employers who insist on their staff changing passwords every 90 days probably aren’t increasing security, and may be making things worse. The same goes for some of the password rules that your bank insists you follow – no more than 12 characters, spaces not allowed, etcetera. At the bottom of all this is the truth that passwords, as a method for keeping our most private data secure on the internet, are fundamentally broken. When I asked the veteran security researcher Bill Cheswick if there was a way to solve the problem once and for all, he thought about it, then suggested, “Burn your computer and go to the beach.” But though the system may be in chaos, there are things you can do to stay both safe and sane. It’s just that they are not necessarily the things you’ve been told.

Password hacking takes many different forms, but one crucial thing to understand is that it’s often not a matter of devilish cunning but of bludgeoning with brute force. Take the example of a hacker who sneaks on to a company’s servers and steals a file containing a few million passwords. These will (hopefully) have been encrypted, so he can’t just log into your account: if your password is “hello” – which of course it shouldn’t be – it might be recorded in the file as something like “$1$r6T8SUB9$Qxe41FJyF/3gkPIuvKOQ90”. Nor can he simply decode the gobbledygook, providing “one-way encryption” was used. What he can do, though, is feed millions of password guesses through the same encryption algorithm until one of them – bingo! – results in a matching string of gobbledygook. Then he knows he’s found a password. (An additional encryption technique, known as “salting”, renders this kind of attack impractical, but it’s unclear how many firms actually use it.)

This is where the length of your password makes an almost unbelievable difference. For a hacker with the computing power to make 1,000 guesses per second, a five-letter, purely random, all-lower-case password, such as “fpqzy”, would take three and three-quarter hours to crack. Increase the number of letters to 20, though, and the cracking time increases, just a little bit: it’s 6.5 thousand trillion centuries.

Then there’s the question of predictability. Nobody thinks up passwords by combining truly random sequences of letters and numbers; instead they follow rules, like using real words and replacing the letter O with a zero, or using first names followed by a year. Hackers know this, so their software can incorporate these rules when generating guesses, vastly reducing the time it takes to hit on a correct one. And every time there’s a new leak of millions of passwords – as happened to Gawker in 2010 and to LinkedIn and Yahoo this year – it effectively adds to a massive body of knowledge about how people create passwords, which makes things even easier. If you think you’ve got a clever system for coming up with passwords, the chances are that hackers are already familiar with it.

The least hackable password, then, would be a long string of completely random letters, numbers, spaces and symbols – but you’d never remember it. However, because length matters so much, the surprising truth is that a longish string of random English words, all in lower case – say, “awoken wheels angling ostrich” – is actually much more secure than a shorter password that follows your bank’s annoying rules, such as “M@nch3st3r”. And easier to remember: you’ve already formed a memorable image of some noisy wheels waking up an ostrich fishing by a riverbank, haven’t you? As the popular geek comic XKCD put it last year, making exactly this point, “Through 20 years of effort, we’ve successfully trained everyone to use passwords that are hard for humans to remember, but easy for computers to guess.”

It gets worse: because passwords are too hard to remember, we’ve added account recovery processes involving security questions that are far too easy for the hackers to answer. That’s how Sarah Palin’s personal email was hacked in 2008: the intruder correctly guessed information about her postal code and high school. A related weakness in account recovery was also to blame for a vicious hacking assault on the Wired magazine writer Mat Honan in August. Hackers managed to commandeer his Google account, send racist messages under his name on Twitter and remotely wipe all the data on his laptop, phone and iPad. All this happened, one of the hackers later told Honan via online messages, because Amazon’s customer services line was happy to give out the last four digits of his credit card number – which was what Apple’s customer services needed in order to reset access to his Apple iCloud account.

Some websites will let you use a passphrase, like the one about the angling ostrich. But many won’t – and in those cases, several security experts agree, you should defy your bank and write them down. Their logic is simple: when you know you can’t commit something to paper, you keep it simple, so you end up choosing insecure passwords. (The same applies to the advice – sometimes a requirement – to change your password regularly: the more passwords you have to remember, the more pressure to choose easy ones.) “I have 68 different passwords,” a Microsoft security specialist named Jesper Johansson told a conference several years ago. “If I am not allowed to write any of them down, guess what I am going to do? I am going to use the same password on every one of them.” The cryptographer Bruce Schneier, another advocate of writing down passwords, points out that most of us are pretty good at maintaining the security of small scraps of paper. Whether you can trust your spouse, or your housemates, is the kind of security calculation you’re qualified to make. Whether your bank account might be at risk from a Russian hacking collective really isn’t.

When I put this argument to Neil Aitken, a spokesman for the UK Payments Council – which oversees, among other things, the inter-bank transfers system and the Link network – he did a remarkably good job of remaining calm. The problem, he explained, is that the laws on fraud impose certain responsibilities on bank customers. If somebody swipes money from your account, you’ll have a harder time getting it back if you’re deemed to have been “grossly negligent” in protecting your passwords. “You could have the most difficult-to-interpret password in the world, but if you tell someone else what it is, you’ve blown it.” The council strongly advises British consumers never to write down or share their passwords.

Both sides have a point. That’s the problem with security: it’s always a matter of trade-offs. More convenience means less security; more protection from remote attacks means less protection from a sneaky housemate. Would you rather run a slightly higher (but hard to quantify) risk of losing your money, or condemn yourself to years of password-related hassle? It’s a question almost as perplexing as, “What’s your least favourite car?”

Bill Cheswick – “Ches” to his friends – is far from alone in believing that we’re descending, as a society, into password chaos. What makes him unusual is that he’s willing to accept responsibility for being partly to blame. In 1994, as a member of AT&T’s fabled research division, Bell Labs, he co-wrote a book with the evocative title of Firewalls And Internet Security: Repelling The Wily Hacker. (He also coined the term “proxy server”, one of several things that makes him, in internet circles, a minor deity.) The book helped lay the foundations for modern online security. But now, he says, when we meet in a Manhattan coffee bar, passwords have become “a pain in the ass! Who can keep track of all these things?” It’s a subject on which Cheswick, a volubly enthusiastic man at most times anyway, grows so animated that people at other tables start to look up from their laptops. “And all these rules! You have to mix symbols, cases, numbers…”

Cheswick calls these “eye-of-newt” rules because they resemble recipes for magical potions, although sometimes, when getting carried away giving speeches, he has been known to call them “password fascism”, too. “I have 25 different accounts, so now I have to remember 25 different eye-of-newt passwords? That’s not gonna happen!”

Besides, he argues, the focus on making passwords more complex is rapidly becoming irrelevant, because the more serious threat is from keyloggers – software surreptitiously installed on your computer, via the internet, that monitors the keys you’re pressing. “I don’t care how good your password is – if I’m watching you type, I got you,” he says. You can reduce the risk by using a Mac, or by upgrading from the insecure Windows XP to Windows 7, and installing anti-virus software. But the only real guarantee is never to visit the kinds of sites where such “malware” resides. And “if the grandkids come round, and the teenage son types in one bad address? You’re done.” Hardly less sinister are “phishing” attacks, the topic of much media hype, in which an email or website that resembles something reputable, such as the log-in page of your bank, tricks you into parting with your password. (The basic anti-phishing advice is to check your browser address bar; to hover over links with your mouse to make sure they’re what they claim to be; and never to provide your password in response to an email.)

One day, we may not have to worry about any of this: there are innovations in development that might replace passwords entirely. Touchscreens could be configured to detect subtle aspects of your interactions with your computer – the distances between your fingers, the speeds at which you tap and scroll. Technologists at Rutgers University in New Jersey have built a prototype of a ring, worn on the finger, that would send tiny bursts of electricity through the user’s skin to the screen, vouching for his or her identity. Fingerprint readers, built into some laptops already but with too many flaws to be taken seriously, could be improved. But don’t hold your breath. “Passwords aren’t going away” for the foreseeable future, Cheswick says. “I can wish as hard as I want, but they’re just too convenient.”

In the meantime, he recommends doing what I did, after thoroughly scaring myself researching this article: install a piece of software known as a “password wallet”, such as LastPass or 1Password. These generate fiendishly random passwords for each of the sites you visit, storing them all behind a single master password. I installed LastPass and chose a fairly long sequence of English words with digits. I am now in the disorienting position of not knowing, and never having known, the password to my email, for example, but it doesn’t matter: LastPass provides it whenever it’s needed.

It’s not a perfect solution. LastPass is secure to an almost problematic degree: since it conducts all its encryption and decryption on users’ own computers, my master password is unknown to the company, which means no one will be able to help me should I forget it. (There’s no recovery process based on security questions, either.) And so – yes – I’ve written it down, in coded form, on a scrap of paper, which I’ve carefully hidden. I hope to have the password memorised soon. There’s no such thing as total security, let alone total security plus total convenience, but this feels like a workable compromise. I’d just better not forget where I hid that piece of paper.