Only collecting 10-20 minutes of data

published by Charbucks C on Wed, 01/18/2012 - 13:32
English

I edited this topic name, as I've figured out a bit more, but I'm still having trouble.

I just got my cable and started trying to capture raw data. Right now I'm just using Chad Joan's CSV Script to record raw data (thanks Chad!), and I'm working on a wxPython interface to do more. However, after about 10-20 minutes of recording, I get a series of errors (usually capture errors) and BaseLink.py eventually terminates. Since Chad's script is a simple while loop, this causes the entire program to quit.

Specifically, I'm getting the following error series of errors:

Capture error: bad checksum
Capture error: bad checksum
Capture error: bad checksum
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run()
File "/usr/local/lib/python2.7/dist-packages/ZeoRawData/BaseLink.py", line 136, in run raw = self.ser.read(datalen)
File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 449, in read buf = os.read(self.fd, size-len(read))
OSError: [Errno 11] Resource temporarily unavailable

I figured this was due to a timeout, so I tried adding the following function to BaseLink.py, and I replaced all the calls to self.ser.read with this new self.tryRead function:

def tryRead(self, num):
    """Function to attempt reading and handle errors."""
    try:
        raw = self.ser.read(num)
        return raw
    except OSError:
        self.error('OSError while trying to read serial data. Retrying...')
        time.sleep(0.5)
        self.tryRead(num)
    finally:
        return '0'

A bit of a hack, but it stopped the program from quitting with OSErrors. However, I still can't record more than the first 10-20 minutes of a night - except now, it just exits without giving me any errors!

Do I have trouble hardware? I haven't been able to find reports on any similar problems. FWIW, part of the reason I'm playing with the RDL is because Zeo has become very wrong at recording my sleep since I moved to a new apartment (I know, it doesn't make much sense, but it's the only thing that's changed). Customer support has sent me new sensors and even a new transmitter in an attempt to fix the problem, but so far nothing has improved - in fact, it's getting worse, and now records me as being in deep sleep as soon as I put the headband on, then seems to have random results throughout the night. The only consistent thing seems to be that when I'm awake (both at night and in the morning), it records me as being in deep sleep.

Forums:

Charbucks C

Thu, 01/26/2012 - 15:18

Bumping myself here...

I find it hard to believe that I'm the only one having this problem.

Pj

Fri, 01/27/2012 - 02:51

Maybe your new place has a lot of interference of some sort.  If you can, you might try sleeping at the opposite end of your apartment to see if that makes any difference.  I've also had problems when I'm too sweaty (maybe your new apartment is warmer than the old one) which seems to make the Zeo think that I'm in REM sleep no matter what.  If that's the case, set up a fan so that your forehead doesn't get too moist. 

No ideas about the errors with the code as it isn't anything that I've used.  If by chance you use Linux, I have some code you can try instead:  http://www.ecstaticlyrics.com/secret/zeo_raw_data_parser.tar.bz2

Charbucks C

Wed, 02/08/2012 - 10:50

I was notified that you replied weeks ago, but it took a while before your post actually showed up.

I think you might be right about the interference - when I put the headband on and look at the frequency response, I get almost a perfect negative slope ramp, decreasing in amplitude as frequency increases. The same occurs if I put the headband on, say, my shin, which I figure is a pretty good control for what the head feels like physically without the brainwaves. Unfortunately, I can't move to the other end of the apartment, but if I could get the connectivity problem sorted out, then I could probably filter out the interference and get some more accurate results.

Based on this post (http://www.myzeo.com/sleep/serial-port-gave-ghost-and-im-giving-now), I'm guessing my connectivity problem is hardware related. It's really rather unfortunate. Like Chad, I'm inclined to switch to the Android version, but I'm also loathe to spend the money after buying Zeo bedside and an overpriced cable (plus waiting 6+ weeks for it to ship to Canada).

I haven't yet tried your C version of the code, but I'm not very hopeful given the (likely) hardware cause. However, it may be worth one more try before I give up on this altogether.

Pj

Sat, 02/18/2012 - 22:02

Not having the cash to purchase a new set of headbands, I tried hand-washing mine with some laundry detergent a week ago.  The next night I noticed I had been wearhing the headband for about 15 minutes yet the Zeo hadn't recorded anything, and wasn't showing the "head present" indicator.  Wiggled it around a bit and it worked, so I went to sleep.  Few days later the same thing happened, but then it occured to me that the headband was probably just getting too worn out.  So I swapped it out for a new one (sadly my last) and it seems to work fine now.  Don't know how old your headband is or anything, but it seems worth a mention just in case the move was purely a coincidence.

As for that serial port...

I'll be the first to say that the connector doesn't snap in place, and that wiring that thing up is complex from the point of view of the average person, as is dealing with any serial port, but with all of that aside, it just isn't a big deal either.  I've used mine for seven months without any problems.  ...of course, I also wrote my code so that it doesn't bomb on the occasional data error.

...and really, I'd be surprised if that was really the problem with the code you're using.  Seems like a really dumb mistake.

Maybe the actual problem is that your USB cable is coming loose, on the side where it connects to your computer.  Thus, the serial port completely disappears from the computer, and further attempts to read from it result in that "resource temporarily unavailable" error.

I always wonder why the USB designers didn't seem to consider that problem.  Plug in an external hard disk, write some data, and just slightly nudge the USB cable enough to disconnect it from the PC for a millisecond, and in that millisecond the PC has enough time to think "OMG, that drive is gone!  Well, let's just toss out all of the unwritten data in the cache..." followed by "Oh look!  Someone plugged in a new USB disk drive!  ...and, OMG, someone failed to 'safely remove drive,' so let's start up the file system repair process..."  Seems to me that the standard is broken by simply removing devices that are unplugged while in use, rather than popping up a dialog box with a message to the effect of "would you like to plug that back in, or just crash this program that was using it?"

I've also had problems in the past with a bad USB hub.  For about a year I fought with what I thought was buggy FTDI drivers in the Linux kernel, which exhibited all sorts of foul behavior, sometimes even requring me to reboot if I tried to do something as simple as write more than a certain number of bytes in a sinlge write() call.  With every kernel version the rules about what would or would not mess up my system state seemed to change.  Eventually I figured out the problem had nothing to do with the kernel at all.  If I just removed the USB hub I had between the FTDI chip and the computer, the problems went away and everyhing was just fine.  Never had a problem with anything else attached to that hub, just the FTDI chips.  

So I definately wouldn't give up too soon.  Baud rates like 38400 are so ridiculously slow by modern standards that the idea that there is going to be any sort of error is just silly.  Also, the CMOS-style logic levels are far more reliable than what was used on serial ports in the past, with cleanly high and low voltages on outputs and virtually non-existant current requirements on inputs.  CMOS logic levels are basically the ideal of what circuit designers always wanted, but were forced to use other garbage in the past.  So as long as your connector isn't about to fall out, you shouldn't be having any problems at all.  I'd look towards the USB end for the problem.

As for the decay in the frequency analysis that you've seen, that may just be what it looks like.  I analysed the frequency response of my Zeo a while back, and came up with this graph:  

http://www.ecstaticlyrics.com/pinnwand/messages/adventures_in_signal_pro...

As for the raw data, it is basically just a bunch of squigly lines.  I don't know how the Zeo makes any sense of it honestly.  I've seen people post screenshots of what they think are sleep spindes and K-complexes and other features, but I'm not convinced that anyone has seen anything other than random noise.  I once built my own EEG, and with proper electrode placement you can get some easy-enough to interpret data.  For example, the alpha waves on the back of your head that show up when you close your eyes while awake are so obvious a child would spot them without even knowing what they're looking for, and with that, spotting any awakenings during the night is bloody simple.  However, with eletrodes on the forehead...  I think there's a reason that sleep researchers don't bother to put electrodes there.  The sinus cavities underneath make the signals on the forehead virtually worthless.  That the Zeo is able to take those worthless signals and determine a sleep stage from them seems like magic.

Nevertheless, I too have been wanting to analyze the raw data myself hoping to get something out of it that's more relevant to me, even if it isn't anything that's relevant to anyone else.  Just haven't had the energy to really try yet.

Charbucks C

Mon, 02/27/2012 - 23:35

Thanks for that very detailed response, Pj. I've actually tried a few different headbands, and even a different transmitter, thanks to Zeo support, but nothing seems to solve the problem. It really seems to be entirely on my end. The annoying part is, every now and then it will work just fine - it records me as awake when I'm awake, and a reasonable amount of light/REM/deep (i.e. not 60% deep). That's part of why I'd like to access the raw data - just to get to the bottom of things.

I think you might be on to something with the USB connection, though what could be knocking it loose is beyond me. I can try a different computer and see if that helps, as currently Zeo's hooked up to an old Eee900 that's been tossed around a bunch. I'll try it out with a more reliable (or at least clean) computer and see if that doesn't fix it.

Your frequency response certainly does seem to correspond to what I was seeing. All this would be much easier to diagnose if we knew Zeo's magic algorithm for determining sleep state...

Pj

Tue, 03/13/2012 - 00:02

I had my problem again, the one which I thought was caused by a dirty headband. I noticed that, in the live data, I was getting an "invalid impedance" signal, and no raw data since it only sends raw data when it likes the impedance. Also on the Zeo display, it simply displayed the headband indicator, without the head that it lights up under the headband after you've put the headband on your head.

I decided maybe my forehead was too dry, causing poor contact between the electrodes and my head, and so it didn't think I was wearing the headband. So I licked my fingers and then smeared the moisture on my forehead. It worked after that.

That it would be due to dry air makes a lot of sense for me since my new house is a lot dryer than my old place. I frequently notice static charge on various things, and sometimes when my hair is nice and clean, it'll do the "Van de Graaff generator" thing when I comb it. I should probably run a humidifier. So you might try adding some moisture to your forehead before going to sleep. Try some plain water, or perhaps some moisturizing lotion or baby oil. Just make sure that whatever you use is good and wiped off before you put the headband on so that you don't dirty it unnecessarily.