TechHawks.org

I’m going to preface this post by saying that this is a topic that I’m absolutely passionate about, but I realize that it’s something that most educators have not considered deeply if at all.  The process of sending electronic data to the State DE has an extraordinary impact on everyone at Prairie. But, up to this point it’s pretty much an invisible process.  It does take a great deal of resources to make it happen, and it touches us all.  I really think we are at a critical juncture with our relationship with the State DE on how this process works.

What I’m suggesting is that all schools in the state of Iowa should be compelled to use the same student information system.  We use Infinite Campus.  While I think Campus is the best fit in that they have a proof of concept that their product works for large scale deployments (they are used state-wide in Kentucky, Maine, South Dakota, and Wisconsin), I would support switching to another product if everyone else in Iowa did so too.

I see two paths ahead.  One route is visionary, highly efficient, but politically more difficult to achieve: a state-wide student information system.  The selection process should be open and transparent.  A state-wide system would allow both the DE and local districts for focus more resources (time, talent, and treasure) on getting results with students instead of allocating resources required the process-overhead of moving data between different systems.

The other path is one that is the one we are currently on: local control of student information systems and integration of the data at the State DE.  This is highly inefficient, resource intensive, but is politically easy to achieve.  It forces local districts to divert resources from kids in order to ensure that accurate data is sent to the state.

I’m not alone in my belief that the State needs to look at a new model for managing student data.  At the last UEN meeting (Urban Educator’s Network – a group of the 18 largest districts in the state), the Technology Directors’ job-alike group drafted the follow resolution to Jason Glass

We believe it is an inefficient use of state financial resources for each district to maintain their own student information system. Statewide network infrastructure has reached the point that we feel it is time for a statewide, centrally hosted, student information system.

The core benefits of a state-wide student system would be reduced cost for both districts and the State.  This assumes that some sort of volume licensing agreement could be reached with the company providing the product.   So, it would cost local districts less.  The State would be able to spend less on personnel to support the data exchange process (more on that below).  This type of setup would also give both the State and districts faster, more efficient access to student data.

The screen shot above is directly from a presentation from the DE on the central role that Student Information Systems (SIS), i.e. Infinite Campus, plays in school districts.  All of the white boxes represent data collections from the DE.  Wouldn’t it be so much simpler if we were all using the same program?

The UEN resolution has been nearly unanimously endorsed.  It’s a step in the right direction.  But, it probably won’t be enough.  Last December, I was invited to be part of an advisory group to help set the educational technology agenda for the 2014 Iowa legislative session.  I advocated strongly for this idea with Jay Pennington and Vic Jaras, the DE representatives.  My guess is that it will take a grassroots effort as well to get this done.  I would encourage each teacher a Prairie to contact your local state representative and senator and promote this idea.  I would think this would be a very appealing notion to legislators on both sides of aisle:  an idea that is good for kids and saves the state money.

Here is some additional context.  Read on if you want to see how “the sausage is made” for getting data to the State…

When I started at Prairie in 1999, I worked with the high school guidance office to electronically submit our student data file for Project EASIER.  EASIER was the unfortunate acronym the Iowa D.E. used to describe student data transfers from local schools to the DE during the fall and spring.  As the years went on, the EASIER acronym became the very definition of ironic.

At that time, fourteen years ago, we submitted data just for high school students.  Each student had thirteen data elements associated with them.  So, for the entire district, that was roughly 15,000 data points submitted twice a year.  The data itself was frankly not all that important – post-graduation intentions for exiting seniors, etc…  However, with the ascension of No Child Left Behind, Project EASIER began to grow in scope.  The total district student population was required (no longer just high school aged students).  Numerous data elements were added (well over 120 per student now) and an additional collection period (winter) was put in place.  So, that means we submit well over 750,000 distinct data elements to the State three times a year.  Each file must come out of Campus completely error free.  If there’s one error, we must locate it Campus, correct it in Campus, extract the 750,000 data elements again, and resubmit the file to the State site (and keep our fingers crossed that no one across the rest of the district has changed something else in Campus that may result in an other error).  Sounds kind of painful, right?  It is.  In fact, I don’t think the DE could even say the name Project EASIER with a straight face anymore.  So, last fall they rebranded it to be SRI – State Reporting in Iowa.

Our three annual SRI submissions (fall, winter, and spring) filter data into the following DE accountability systems:  Certified Enrollment – this program calculates the per student funding for the district; AYP (Adequate Yearly Progress) – this program calculates each building’s No Child Left Behind status; Graduation Verification – this program determines the district’s graduation rate; and CTE/PSE0 (Career and Technical Education/Post Secondary Education Option) – these programs determines supplemental funding for career tech and reimburse us for all the classes students take at Kirkwood or other higher ed institutions. But, the real kicker is that if error is found in any of these “feeder” applications (CE, AYP, CTE, PSEO), which are often not even available to look at until weeks after SRI certification is done, we need to uncertify the SRI file, fix the error in Campus, re-generate the SRI extract (750,000 error free data elements), resubmit it to the SRI application, recertify SRI, then look at the feeder application to see if error is still there or not.  Again, Campus is an ever-changing system, so it’s always possible something else has changed in resubmission process to cause a different error.

Just to give an idea of what’s at stake…  Missing just one student in Certified Enrollment costs the district over $6,000.  Having an incorrect enrollment date for just one student can mean that AYP is off for a building – that can mean the difference between being on or off the Building in Need of Assistance list.  In addition to all of this, the State is also moving toward a total electronic exchange of transcript data between high schools and state regent universities that will come from our various local student systems.   Clearly, over my time with the district, electronic record submission to the State DE has gone from being a low impact and low accountability item to being arguably the highest impact and highest accountability activities in the district.

Not all data related problems go away with a state-wide system.  We’ll replace on set of difficulties with another.  But, the path we are on right now is unsustainable and worst of all it’s bad for kids.  The DE’s current response to my type of objection is SIF.  That’s a completely different rant, and I won’t go into that here.  But, in my opinion, SIF will add another layer of complication to an already highly complex process — further accelerating the inherent unsustainability of our current system.  It’s time to simplify.  It’s time do what’s right for kids.  It’s time for one student system for the State of Iowa.

First of all, I’d like to thank everyone at Prairie for being so understanding last week.  It was a tough time.  But, the people here were so great.  I hope it was one of those “once in ten years” type of events. So, many people are understandably curious to know exactly what happened.  So, here, a rather detailed, play-by-play if you are interested.

I came into work on the morning of Tuesday, April 9^th knowing that we had some sort of network service disruption.  Kathy Waychoff had called me at 6:00ish AM.  I arrived onsite around 6:30.  The previous night we had some significant storms, so I assumed that the power had gone out overnight.  When we lose power for over 30 minutes our emergency backup power drops and everything is shutdown.  When power is restored, it almost always requires some human intervention to come back up correctly.  While this is inconvenient, it’s not that big of a deal.

However, this particular morning was different.  I walked into the data center that houses our servers and everything else the runs the network to see water on the floor.  This, of course, got my attention quickly.  The puddle was in a neat semi-circle around the main enclosure.  There was not water under it.  I looked up and saw a slow, (1 drip every 5 seconds or so) but steady drip of water coming from the air conditioner in the ceiling.  I grabbed a chair and looked at the top of the enclosure to see how much, if any, water had accumulated there since the puddle did not extend under the cabinet.  But, to my relief, the top of the cabinet was totally dry.  In fact, it still had a coating of dust.  There were no splatter marks or anything.  So my initial thinking was that we dodged a bullet:  water only in front, none on top, nothing underneath.  It looked like all of the water had miraculously missed everything.

I then scanned the equipment in the enclosure.  Here’s what I saw from top to bottom…

• Firewall: good – green lights
• Router: good – all green lights
• Core Switch (connects all of our buildings together): good – all green lights
• iBoss Reporter (a device we use to monitor network traffic) – good  — all green lights
• iBoss Primary Filter: good — all green lights
• iBoss Secondary Filter: good – all green lights.
• SAN (storage area network – a big cluster of hard drives that store all the district’s data): good – all blue lights
• SAN control station (computer that monitors the SAN): good – all green lights
• Fiber Switch (connects the SAN to the core switch): good – all blue lights
• Infinite Campus Sever: good – all green lights
• VM Host 1: offline – amber light
• VM Host 2: offline – amber light
• Battery Backup 1: good  –all green lights
• Battery Backup 2: good – all green lights

At this point, I still was not very concerned.  Typically the VM hosts don’t always start up correctly with out someone there.  This would often be how things would look after a power outage.  I then inspected the scrolling text flashing in amber lights on the front of these two servers and saw “…failsafe voltage…”  This got my attention.  Again, I was not overly concerned at this point.  If you look at my list above there are ten devices above these two servers in the rack and two below them that are all working.  There was no water on top of the rack and none underneath.  It didn’t seem very likely to me that water was the cause of the problem.  It just didn’t seem possible that water could miss the 10 pieces of hardware above and the two below them.  I thought that perhaps there was a power surge from the storm the previous night that caused the problem.  I optimistically thought we might just need to reset something in the start up process to get this these two computers back online.

The two VM hosts are crucial hardware – “the brain” of our whole system.  We have two just in case one fails.  We also purchased a five year, four hour parts and support agreement with Dell for them.  This means if we have a warranty related hardware failure, Dell will get parts here in under four hours.   Without these two severs, none of the rest of the network can function.  They host our DHCP, DNS (both necessary for internet connections), email, web server, food service server, library server – pretty much everything.   All of the data for each of these servers is stored on our SAN.  But, with out the servers on the front end, our SAN was like a giant USB flash drive with out a computer to plug into.

When I could not get the VM host hardware to do anything, even after trying different power sources (I thought maybe a circuit breaker had gone or something like that), I called Gary Sicard, our external hardware support consultant.  But, it was still before seven AM, and he was not answering his cell.  So, I pulled up Google on my iPhone and searched up the error message scrolling on the front of the  servers.  My worst fears are confirmed.  All the resolutions involved replacing all internal components that touched a power source – basically everything!  For me this was the “Houston we have a problem…” moment.  I knew we were in deep trouble.

After calling Duane Carver at Ops, I went straight to John Speer and Steve Doser to develop a plan.  They were great.  They started working on messages to communicate with buildings and the public to let them know we were without services would be for at least a few hours.  By the time, my staff were starting to come arrive.  I had Collin start a support call to Dell.  From the outset, I had a bad feeling about how this call would go.  I knew if we could claim this under warranty, we’d have parts here before 3:00 PM and could probably have everything restored by the following morning.  But, I was worried about how the Dell folks would view the possible the water with the warranty.

My worst fears became reality.  While on the support call, Dell did determine that it was water that caused the problem.  At this point, the call turned from a warranty/support call to a sales call.  We spent the next 5 hours on the phone with Dell trying to get parts.  Dell policy will not allow their sales people to use the warranty repair parts depot.  And, because our two servers were over three years old, it was hard to find parts.  We started hearing scary things like, three business days, five business days, or worse – unknown.  It was maddening!   I knew Dell had everything we needed in their parts depot in Davenport.  But, we couldn’t access it because this was a non-warranty repair.  Finally, at the end of the day, around 3:30 PM, we got assurances that half of the parts would arrive on Wednesday morning with the rest come on Thursday morning.

While all of this was going on, the rest of the team was working to make the best of what we did have available to us. Bill Paarmann worked to get our web domain (www.prairiepride.org) redirected to the district Facebook page.   I was on the phone with Campus desperately trying to get Campus Messenger to work.  The Campus support team was incredibly helpful.  I ended up working with one of the main hosting engineers via his cell phone on his day off.  With his help, we did get a message out.

Our local maintenance folks had been working hard on this problem, too.  They were looking at the cause of the leak.  After draping all of the servers in plastic, they began pulling apart the air conditioner.  Since we don’t have many of these units on campus, they soon had to call in the cavalry, too.  Wednesday afternoon we had three techs from Colony Heating and Cooling working in the ceiling.  It was a defective pump at the center of all of this.  The part had to be ordered for that as well.

While the damage had hit us with the uncanny precision of a laser guided smart bomb, we did still have a lot of stuff that worked.  I wanted to capitalize on that.  So, this began our “Apollo 13” moment.  Charlie, Chris, Nicole, and Collin worked to coble together something to get us back in the game.

They found an old server in our parts closet, and took RAM (memory) from our broken hosts and loaded as much as the old server would accept (16 GB).  The VM Ware OS and configuration are stored on redundant SD cards in the main hosts (the same sort of simple memory cards used for older digital cameras).  They barrowed a USB SD card reader from Kathy Waychoff, and plugged it into the old server.  Unbelievably, it worked!    We had just enough “horsepower” to bring up one DNS/DHCP server to get internet connectivity working.  And, there was just enough resources left over to launch the district web server.  Once that was done, we had to undo the Facebook redirect.

That was our Tuesday, April  9^th.  On Wednesday morning at about 10:00 AM I got a call from our local Dell reseller.  He had the unfortunate chore of letting me knew that the Dell sales team had goofed.  The first shipment of parts would not be arriving until Friday and they didn’t know when the rest would be coming.  While this was not his fault, I have to admit I did “shoot the messenger” a bit.  I won’t go into the specifics here, but before that day was out this reseller had arranged to get all the parts we needed here (with Fed Ex tracking) by 8:00 AM Friday.  Ironically, we didn’t use Dell at all, we went with Fox Micro – another reseller.

On Thursday, we received about half of the parts necessary.  Not surprisingly, this handful of parts did not correct the problems.  We had to wait for the Friday shipment to know.  The rest of the parts were scheduled to arrive before 8:00 AM on Friday.  However, sever storms on Thursday night had delayed the Fed Ex shipment until 11:00 AM.  Once the parts arrived we set to work replacing everything on the inside of these to servers.

At first, it went fast and well.  I was confident we would have all services restored before kids left for the day.  After all, we had run the district off of the cobbled together server to three full days.  It should be no problem to get the main hosts working again…  And, this type of thinking was my down fall.  I won’t go into excruciating detail on the rest of the day Friday.  But, there were many, many problems getting these two hosts to come back online correctly.  I locked the door to the ESC at 1:30 AM Saturday morning.  Even though it had been a long, long day, I left feeling great.  We had restored all services.

Someone told me later, after this was all over, that no one knows what it’s like to be a teacher unless you are one or are married to one – the same goes for technology people, too, I think.  I realize this post is incredibly self-indulgent.  I’ll be shocked if anyone actually reads it end to end.  But, it was such an interesting set of events.  I thought it was probably worth relating what exactly happened as very little of the activity was visible to anyone outside of the central office.