Announcing Underscore Backup 2.0 and service general availability

First stable version of Underscore Backup with support for the companion service is now available. The service itself is also generally available.

Even with the new service, the main focus of the application is privacy, resiliency, and efficiency. The new service does significantly simplify setting up cloud backups and sharing though compared to use

The main new feature in version 2.0 is the introduction of a companion service that will help with many aspects of running Underscore Backup such as.

• Keep all your sources organized in one place to easily restore from any of your backups to any other backup.
• Help facilitate sharing of backup data with other users.
• Optionally allow private key password recovery.
• Easily access application UI even if running in a context where a desktop is unavailable, such as root on Linux.
• Use as a backup destination. Storing backup data is the only feature that requires a paying subscription, giving you 512GB of backup storage for $5 per month.
• Support multiple regions of data storage supporting North America (Oregon), EU (Frankfurt), and Southeast Asia (Singapore) regions to satisfy latency and data governance requirements.

On top of the companion service changes, the following features and improvements have also been implemented.

• Added support for continuous backups by monitoring the filesystem for changes.
• Introduced a password strength meter which requires a score of at least “ok” when setting up.
• Switched from pbkdf2 to Argon2 for private key hashing function.

On top of the companion service changes, the following features and improvements have also been implemented. Get started by downloading the client now.

Launching Underscore Backup service and first beta of the version 2.0 of Underscore Backup application to use it

Finally launched the first public release of the Underscore Backup service. A backup service that is a companion to the Underscore Backup application I have now been working on for a few years. I am really excited about this since adding a service component to the application solves a couple of user pain points with my previous releases, such as.

• Its now easy to coordinate and keep track of multiple sources so you can easily restore data from another computer you are backing up.
• It comes with the ability to use the service for storage so that you don't have to deal with configuring S3 or something similar. The storage also supports 3 regions in the US, EU and the Asia Pacific region and is priced lower than S3.
• It makes it easy to set up sharing between users.
• The service can provide optional secret key recovery.
• You can easily keep track of where the administration interface of the application is available even if the application is running in a context that does not have access to your desktop.

There is also a ton of other additional features included such as improve password hashing algorithms, log rotation, integrated password strength meter and built in new version notification.

You can head to the site to download the latest version and sign up for the service. The service is entirely free, however you do need a subscription for using storage as detailed on the pricing page.

My quest for fiber provided by AT&T

© Can Stock Photo / kasezo

When I moved to a new house about two years ago, I was disappointed to learn that there were no options for fiber-based internet in the area so I would have to take the step down to cable based internet. Fortunately, I was pleasantly surprised in September of 2020 that AT&T Fiber had added support of my area.

First try in September 2020

I ordered it as quickly as I discovered it, even though I was a bit hesitant about having people in my house as COVID cases were on the rise (I have a person in my household who is in a risk group). The person on the phone with AT&T assured me that all AT&T personnel involved with the install would be wearing a mask though, so I proceeded regardless.

The day of the appointment I was excited and had cleared my schedule. The first person to show up was not the technician, but just a salesperson that wanted to make sure I didn't have any trouble creating my AT&T account (Which I had already set up days before as per the instructions in the AT&T communication). This person also assured me that they get in trouble with AT&T if they do not wear a mask which felt reassuring to me.

About an hour later, still during the appointment window the installation technician showed up (Still within the assigned service window). We tried to figure out where the AT&T connection at my house was and eventually found it. Unfortunately, there was no fiber pulled to my house and it needed to be pulled around 100 feet from a neighbor's access point. He tried snaking the existing conduit but failed. He needed to call in a specialist that both had better snaking equipment and if that failed, they might have to do some digging to fix the conduit.

The second technician made an appointment and showed up around a week later. He also had a helper with him. They spent a good hour trying to get through the conduit. They also failed and I was told that they now would have to bring in a 3rd party company that would try again and might potentially had to dig a new conduit.

The third technician just showed up with no appointment. He also had no mask and did not have one to put on when I asked about it. I told him to come back when he had a mask and at an appointed time. After this interaction I called AT&T to complain and was told that the mask mandate only really applied to AT&T employees and since this person was a 3rd party contractor there was nothing they could do. At that point I told the representative that I only wanted this after I had gotten the expressed promise that everybody involved would wear a mask and since that was not true, I did now want to cancel the order. The AT&T rep told me that they did not have the authority to cancel my order, but instead that had to transfer me to a loyalty specialist. I told them that they could either cancel the order or not but that I would not open the door or let them on my property and hung up.

About a week later another man showed up from AT&T, also not wearing a mask. I told him that, no I had not ordered any AT&T Fiber and that he should go away through the closed door. A week after that an additional person showed up from AT&T, this time with a mask. I explained to him what was going on and he apologized and said that he knew how to make this issue go away for me. And in fact, it turned out that he did because that was the last I heard from AT&T for the time being.

In total this first try involved 7 visits from AT&T with a total of 8 people visiting my house.

Second try in March 2021

Skip forward to March 2021, as I am now vaccinated, I decided it was time to make another try. I also happened on an ad for AT&T Fiber with a good introductory offer, so I decided to try again with an order placed online on March 8th. I got an initial appointment for the morning of March 18th. About an hour after the appointed time with no visit I called AT&T customer support. I was told not to worry. The technician was just running late, and he was still on the way. After another 3 hours I called again and was then told that the technician had gone to the wrong house and since nobody at that house had ordered internet he had left. At no point during this had AT&T proactively reached out to me to let me know what was going on.

Slightly miffed I rescheduled the appointment for a week later. That day came around and no technician showed up that day either. At around 2 hours after the appointment window ended, I called support again. The first person told me not to worry, the technician was just running late. I told them about my experience last time and was put on hold to a second operator. This operator said the same thing. At this point I told the operator that this is no problem. However, if the technician does in fact not show up then they do not have to try again. At this point the agent transferred me again to a "Loyalty Specialist".

This third person that I spoke to did in fact do some digging and figured out that when the technician who had gone to the wrong house and left, he had in fact cancelled the entire installation. And me rescheduling it with the support agent did not actually reopen that ticket so there was no technician coming. He then proceeded to say that I shouldn't worry, he knows how to restart the process again properly. At that point I said "Thank you, but no thank you. You gave it the old college try but couldn't even get a technician to my house in 2 tries so I am done".

Third try using Sonic Internet

I had discovered that Sonic Internet also resold AT&T Fiber at my location and figured that at least in that case I would deal with a support department that was prompt and knowledgeable even though I would still have to deal with AT&T for the actual installation. The same day that I cancelled the second try with AT&T I ordered Fiber from Sonic instead.

The first appointment was scheduled on March 31st. Same as the original visit that I had in September a year earlier the conduit is broken and need to be fixed. This technician managed to get the specialist team to do the second visit the same day though. They showed up as a 2-person crew and told me in refreshing detail what was going to need happening next. First an underground survey needed to be performed after which a digging crew would be dispatched to fix the conduit. I should expect the survey to happen within a few days and then the digging crew would show up in a week or two.

On April 6th I had a second appointment scheduled at which time an AT&T technician showed up to install my internet with the assumption that the fiber had by this time already been installed. Of course, the underground survey had not even happened yet, so he had to leave without anything done.

April 8th 2 big trucks with a team of 5 people showed up. They started by taking an hour lunch and after that got down to the work of digging my conduit. When I pointed out that the underground survey had not yet been done, they got a bit flummoxed and told me that unfortunately they could not do any digging until this had been completed. But the foreman told me that he had put in a rush order to make sure the survey would get done as soon as possible.

On April 12th I got an email from Sonic telling me that they had been instructed by AT&T to check that my internet was working correctly. At this point of course, there had still not been any actual work done by AT&T, so I sent an email to Sonic support letting them know this.

On April 15th I got a notification from Sonic telling me that the installation of my internet had been scheduled for April 19th. Since this sounded strange, I contacted Sonic support to tell than that I am not currently waiting for an AT&T technician, but an underground survey. What I was told is that the last people who were here marked the installation as complete (Which is why I got a notification earlier in the week making sure my internet was working correctly) and because of that they now had to start over from the beginning. Which means that a person must first come out and assess that a dig needs to happen (So starting all the way from scratch again). Was told by Sonic rep that they had gotten into a discussion with AT&T that got so heated that the AT&T rep hung up.

On April 16th I got a visit from a cheerful AT&T customer service rep asking me how I was enjoying my new AT&T Fiber internet. She got an earful of what I thought of AT&T at that moment.

April 19th comes around and I get a visit from another AT&T customer service rep to help me set up my AT&T account. I explain the situation to him, and he promises to get on the phone with his manager to see if there is anything he can do to help. While he does that the AT&T installation technician shows up. The technician asks if I am speaking for Yvonne? I tell him that I have no idea what he is talking about and he tells me that his work order says that he there to install internet for an Yvonne from Florida through the third-party provider Earthlink (Not Sonic). There is literally nothing in the work order that is correct except for my address. I do manage to get the technician on the phone with Sonic support and both escalate the issue to their managers. In the end there is nothing AT&T can do to have the technician do the work, even though he is here. He has to come back at a later date when the order has been corrected. At this point the AT&T service rep steps back and says that he will take me under his wing and sort this out for me. I pointedly ask him if that means that I would become and AT&T customer instead of Sonic. He says yes and I politely refuse.

After AT&T leave, I spend some more time with Sonic support. They promise to get back to me when this is sorted out. While this is happening, on April 22nd, another AT&T technician shows up to do a fiber install for Yvonne of Florida. Later that same day I hear back from Sonic support and they tell me that they have sorted out the issue with AT&T and that I now have an appointment for April 27th (Next Tuesday) to get this process started.

To sum up

So far AT&T have made 15 visits to my house with a total of 21 people. That does not include the visit they made to the wrong house in the second attempt with AT& or the 2 appointments they scheduled for that try. There has been no progress made whatsoever to actually installing fiber and the person that is coming on the 27th is actually the first person AT&T is dispatching for this install from their perspective.

To be continued...

Released Your Shared Secret Service

I recently published Your Shared Secret service which allows you to safely and securely ensure that private information that you have is not lost if you are in any way incapacitated.

The basic premise is that information is submitted through your browser where it is encrypted before it is ever sent to the service. The key to decrypt the information never leaves your browser. The key is then then chopped up into multiple pieces which are securely handed out to a number of people that you chose to act on your behalf and only by a group of them collaborating (You chose how many) can they together assemble the key required to access your information. For a quick introduction you can check out this video.

I really went all out on the privacy aspect of this website and service and have gone out of my way to not collect any information not needed for the operation of the service. The site have no third party links except for when collecting payments and it does not collect any visitor analytics such as for instance Google Analytics.

You have complete control over who of your caretakers is able to initiate accessing the information and also how many of the total group of caretakers need to participate to access the information. Even better the service does not even need know how to contact the caretakers. This information is only known by the unlocking caretaker and the owner of the information.

Furthermore the act of one of your caretakers trying to assemble the key will give you as the creator a notification that allows you to cancel the unlocking information or delete the information all together within a 7 days quarantine period. For more information on how the service works see the Usage section on the website.

The entire service operates on a zero trust model where all the functionality is ensured with cryptographically strong primitives with the single exception being the 7 day quarantine period. There is plenty of detailed information on how the encryption works in detail and how the service has been built. To ensure what is claimed on the site is actually what is happening the source code to the entire service is published on GitHub and you can even run the entire website locally by just cloning the website repository and just run.

npm start

The service is available for an introductory price of $1 or if you want to have completely anonymity you can also pay with either Etherium or Bitcoin although that is slightly more costly because of the value fluctuations of these currencies.

The service is available now so feel free to get started now keeping your information safe without you.

Comparing Macbook Pro to Windows 10 based laptop for software development

My post from a few years ago about Why I hate Mac and OSX is by far the most read post I have ever posted on this blog (Somebody cross-posted it to an OSX Advocacy forum and the flame war was on). So it has been a few years, both OS X and Windows has moved on since 2009 and hardware has improved tremendously. I have also started a job which more or less requires me to use a Mac laptop so I have recently spent a lot of time again working with a Mac so I figured I would revisit the topic of what I prefer to work with.

The two laptops I will be comparing specifically is a Dell Precision 7510 running Windows 10 vs a current 2015 Macbook Pro running OSX El Capitan.

Before I start the comparison I'll describe what and how I use a computer. I'm a software developer that has been working with this for decades. I prefer to use my keyboard as much as possible. If there is a keyboard shortcut, I will probably use it pretty quickly. I tend to want to automate everything I do if I can. I have great eyesight and pretty much the most important aspect a laptop is that it has a crisp high resolution screen (Preferably non glossy) which to me translates to more lines of code on the screen at the same time. So with that in mind lets get started.

Screen

This one is fortunately easy. For some bizarre reason OSX does no longer allow you to run in native resolution without installing an add-on. Even with that add-on installed the resolution is paltry 2880 by 1800 in compared to 3840 by 2160. That means that on my DELL I can fit almost twice as much text on the screen. Also Mac's are only available with a glossy screen which is another strike against it. I don't really care at all about color reproduction or anything like that, and even if I hear that the Mac is great at that (And so supposedly is the DELL) but don't care about that at all.

Windows used to have pretty bad handling of multiple screens before Windows 10, especially with weirdly high resolution. This has gotten a lot better with Windows 10. That said OSX has great handling of multiple screens, especially when you keep plugging in and out of a bunch of screens, things just seem to end up on the screen they are supposed to be when you do. Windows is much less reliable in this sense. That said, the better handling of multiple screens are nowhere near weighing up for the disaster that is the OSX handling of native resolutions or the low resolution of the retina display.

Winner: Windows

Portability

The PC is as a friend of mine referred to it "a tank". It is amazing how small and light the Macbook Pro is compared to everything that they crammed into it.

Winner: OSX

Battery Life

I can go almost a full day on my Mac, my PC I can go a couple of hours. No contest here, the Macbook Pro has amazing battery life.

Winner: OSX

Input Devices

Let me start off by saying that the track pad on the Mac is fantastic. Definitely the best I have ever used on any computer any category. That said why can't you show me where the buttons are (I hate that), the 3D touch feature is completely awful on a computer (I don't really like it on a phone either, but there it has its place). I started this review by saying that I use a lot of keyboard and when it comes to productivity there is absolutely no substitute for a track point. This is that weird little stick in the middle of the keyboard that IBM invented. The reason why it is superior is that when I need to use it I never have to move my fingers away from their typing position on the keyboard so I don't lose my flow of typing if I have to do something quickly with the mouse.

In regards to keyboards both Macbook Pro and the DELL Precision laptops have great keyboards. However, for some weird reason Macbook's still don't have page up and page down keys. And not only are there no dedicated keys for this, there isn't even a default keyboard shortcut that does this (Scroll up and scroll down which are available are not the same thing) so to get it at all you need to do some pretty tricky XML file editing. You also don't have dedicated keys for Home and End on a Macbook Pro. And given that there is so much space when the laptop is open not used by the keyboard on a 15" Macbook Pro I find it inexcusable.

Winner: Windows

Support

With my Windows machine (And this is true for pretty much any tier 1 Windows laptop supplier) I call a number or open a chat and 1 to 2 days later a guy shows up with the spare parts required to fix it. With Apple I take it to the store and then they usually have to ship it somewhere, it takes a week or two... If you are lucky. For me that would mean that I can't work for those two weeks if I didn't have a large company with their own support department to provide me with a replacement to help out where Apple falls short.

Winner: Windows

Extensibility

I can open up my PC and do almost all service myself. Dell even publishes the handbook for doing it on their support site. Replacing the CPU would be very tricky because I think it is soldered to the motherboard, but everything else I can replace and upgrade myself. I also have 64GB of memory, two hard drives and if I want to upgrade a component in a year or two it wont be a problem. The Macbook Pro has Thunderbolt 2 which is great (Although the PC has a Thunderbolt 3 port), but that is pretty much it in regards to self service upgrades.

Also my PC beats the Mac on pretty much any spec from HD speed, size, CPU, GPU, memory.

Winner: Windows

Price

Everybody talks about the Apple tax. I don't find that to be very true. A good laptop (Which don't get me wrong both of these are great laptops) costs a lot of money. And my PC cost quite a bit more than the Macbook Pro did. Granted it has better specs, but I don't think there is really any difference in price when you go high end with a laptop purchase.

Winner: Tie

Productivity

For me productivity is synonymous with simplicity and predictability. Specifically I move around a lot of different applications and I need to be able to get to them quickly, preferably through a keyboard shortcut and I want to do it the same way every time. With that in mind OSX is an unmitigated disaster in this area. First of all, you have to keep track of if the windows you want to get to is in the same application or another one. And if it is another application, you first have to swap to the application you want and then after that you need to use a different keyboard shortcut to find the specific window in the application. I do like that you can create multiple desktops and assign specific applications to specific desktop (Predictable!). However then when you go full-screen with those windows they move to another desktop and this desktop has no predictability at all of where it is placed in comparison to other ones, it is strictly the order in which they are placed. Going on, I still don't understand how OSX still doesn't have a Maximize window button that takes the window and just makes it fill the screen. There are some third party tools that helps you a bit with this madness (Like being able to maximizing windows without going full-screen for instance). And regrettably in my opinion this is an area where OSX is moving backwards where the original Exposé was actually pretty good compared to the current mess. Also I don't like having the menu bar at the top of the screen because it means that it is usually further away from where my mouse currently is which means it takes longer to get there.

Meanwhile Windows 10 in this area took a huge leap with the snapping of windows to the side and allowing you to optionally selecting another window to see on the left. And you can easily switch to any window quickly using one keyboard shortcut same as always

A side note that doesn't affect me much but it does kind of need to be stated is that unsurprisingly Microsoft Office 2016 is just so much better on Windows than OSX.

Winner: Windows

Development Environment

In regards to development environments everything Java is available for both platforms so this comes down to comparing Visual Studio to XCode as far as I think. And obviously this comes down to whether you are developing in Swift or C# but since Visual Studio has recently moved more and more into the multi platform arena this is more of a real choice every day.

XCode has improved in huge leaps and bounds since the original versions I worked with (I started working with it around version 3). However there is simply no contest here. Visual Studio is the best development environment that I know. Both when it comes to native features, and the 3rd party extension system that support it is simply amazing. The only one that might possibly come close as far as I am concerned is IntelliJ.

Winner: Windows

Command Line Interface and Scripting

This is also a very easy call. OSX is Unix based, has a real shell, PERL and SSH installed by the OS. Sure Powershell is OK, but I just don't like it. I would argue that I think the terminal emulation in Putty seems a little bit better than Terminal, but on the other hand it doesn't have tabs and it also isn't installed by default.

Winner: OSX

Software Availability

This is a tricky category because there is obviously a lot more software available on Windows than OSX. However I find OSX has a lot of really good software that isn't available on Windows in similar quality. So I'm going to call this another tie.

Winner: Tie

Reliability

You would think that this is an easy win for Mac. And for normal non power users I would say that is absolutely true. It is harder for a non technical user to mess up an OSX system than a Windows system, no question about it. I however tend to tinker with stuff that normal people wouldn't and I can say that I have managed to mess up my Mac several times to the point where it will not boot and I have to completely reinstall the OS. However, I have done the same thing more times on Windows than on OSX I think. I also am a little bit worried about Apple's general stance on solving security issues in a timely manner, something that Microsoft is actually really good it. That said, even though this is not as much of a slam dunk as you would think I still have to give this to OSX.

Another thing I would like to add in here is that pretty much every PC that I have bought there have been some part of the hardware that did not quite live up the expectations. On my previous laptop DELL Precision m4800 it was the keyboard (In 2 years I replaced it 6 times), on this one I am still working with support on fixing some flakiness with the trackpoint. I have never had similar issues with any Apple computer (Although I did have an iPad 4 where the screen just shattered when I placed it on a table for no reason).

Winner: OSX

Conclusion

If you travel a lot and need to work on battery a lot I think you might want to give the Macbook a go. It's pretty neat.

That said the clear winner for me when it comes to both productivity, usability and just raw performance is going to be a Windows machine when it comes to doing software development. The beauty with Windows is that since there are so many of them you can usually find one that fits you exactly (There are obviously PC:s that are very similar to the Macbook Pro, for instance the bezel-less Dell XPS 15 looks pretty sweet if you are looking for a PC equivalent of a Macbook Pro).

Winner: Windows

How to get the most out of your BizSpark azure credits

BizSpark is arguably one of the best deals on the internet for startups. For me the key benefit that it brings is the 5 x $150 per month of free Azure credits. That said they are a little bit tricky to claim.

The first thing you need to do is claim all you BizSpark accounts and then from each of those accounts claim your Azure credits. This blog post describes this process, so start by doing that.

So after doing this you have 5 separate Azure accounts each with $150 per month of usage. However what we want is one Azure account where we can see services from all of these subscriptions at once and that requires a couple of more hoops to jump through. In the end you will end up with one account where you can see and create services from all 5 subscriptions without having to log in and out the Azure management portal to switch between them.

1. The first step is to pick the one account you want to use to administrate all the other accounts.
2. This is a bit counter intuitive, but you need to start by adding every other account as co administrators to the account from the first step. Yes, I am saying this correctly. All the other accounts need to be added as administrators to the main admin account (Don't worry, this is temporary).
3. The following steps need to be done for each of the accounts except for the main account from step 1.

1. Log into the management console using one of the four auxiliary accounts and go to settings.
2. Make sure you are on the subscription tab.
3. Select the subscription that belongs to the account you are currently logged into. It will be the one that has the account administrator set to the account you are currently logged into. If you have done this correct you should see two different subscriptions, one for the subscription you are logged in as and one from the account in step 1.
4. Click the Edit Directory button at the bottom.
5. In the image below make sure you select the directory of the main account from step 1. It shouldn't be hard because it will be the only account in the list and pre-selected. If you have already set up any co administrators to the account you will be warned that they will all be removed.

6. Add the account from step 1 as co administrator to this account as described in the linked to article at the top of the post.
7. The last step is optional but all the subscriptions will be called Bizspark and hard to keep apart so you might want to rename them.
1. To do this go to the Azure account portal at https://account.windowsazure.com/Subscriptions. This page tend to be very slow, so be patient following links.
2. Click on the subscription name. Your screen might look different depending on how many subscriptions you have.

3. Click on the Edit Subscription Details.

4. Enter the new name in the dialog presented. You can also optionally change the administrator to the account from step 1 at the top, this will remove the owning account as an administrator from the account all together (Although they are still responsible for billing).
4. You can now remove all the other accounts from being administrators to the main account that you added in step 2 if you want.

If you follow all these steps when you log into the account from step 1 you should be able to see all of your subscriptions at the same time in the Azure management console like in the screenshot below.

Keep in mind this does not mean that you have $750 to spend as you want. Each subscription still has a separate limit of $150 and you have to puzzle together your services as you create them to keep all of the 5 limits from running out but at least this way you have a much better overview of what services you have provisioned in one place.

Algorithm for distributed load balancing of batch processing

Just for reference this algorithm doesn't work in practice. The problem is that nodes under heavy load tend to be too slow to answer to hold on to their leases causing partitions to jump between hosts. I have moved on to another algorithm that I might write up at some point if I get time. just a fair warning to anybody who was thinking of implementing this.

I recently played around a little bit with the Azure EventHub managed service which promises high throughput event processing at relatively low cost. At first it seems relatively easy to use in a distributed matter using the class EventProcessorHost and that is what all the online examples provided by Microsoft are using too.

My experience is that the EventProcessorHost is basically useless. Not only does it not contain any provision that I have found to provide a retry policy to make its API calls fault tolerant. It also is designed to only checkpoint its progress at relatively few intervals meaning that you have to design your application to work properly even if events are reprocessed (Which is what will happen after a catastrophic failure). Worse than that though once you fire up more than one processing node it simply falls all over itself constantly causing almost no processing to happen.

So if you want to use the EventHub managed service in any serious way you need to code directly to the EventHubClient interface which means that you have to figure out your own way of distributing its partitions over the available nodes.

This leads me to an interesting problem. How do your evenly balance the load of work evenly over a certain number of nodes (In the nomenclature below the work is split into one or more partitions) which can at any time have a catastrophic failure and stop processing without a central orchestrator.

Furthermore I want the behavior that if the load is completely evenly distributed between the nodes the pieces of the load should be sticky, meaning that the partitions of work currently allocated to a node should stay allocated to that node.

The algorithm I have come up with requires a Redis cache to handle the orchestration and it uses only 2 hash tables and two subscription for handling the orchestration. But any key value store that provides publish and subscribe functionality should do.

The algorithm have 5 time spans that are important.

• Normal lease time. I'm using 60 seconds for this. It is the normal time a partition will be leased without generally being challenged.
• Maximum lease time. Must be significantly longer than the normal lease time.
• Maximum shutdown time. The maximum time a processor has to shut down after it has lost a lease on a partition.
• Minimum lease grab time. Must be less than the normal lease time.
• Current leases held delay. Should be relatively short. A second should be plenty (I generally operate in the 100 to 500 millisecond range). This is multiplied by the number of currently processing partitions. It can't be too low though or you will run into scheduler based jitter of partitions jumping between partitions.

Each node also should listen to two Redis subscriptions (Basically notifications to all subscribers). Each will send out a notification that is the partition being affected.

• Grab lease subscription. Used to signal that the leas of a partition is being challenged.
• Allocated lease subscription. Used to signal that the lease of a partition has ended when somebody is waiting to start processing it.

There are also two hash keys in use to keep track of things. Each one contains the hash field of the partition and will contain the name of the host currently owning it.

• Lease allocation. Contains which nodes currently is actually processing which partition.
• Lease grab. Used to race and indicate which node won a challenge to take over processing of a partition.

This is the general algorithm.

1. Once every time per normal lease time each node will send out a grab lease subscription notification per each partition that.
• It does not yet own and which does not currently have any value set for the partition in the lease grab hash key.
• If it has been more than the maximum lease time since the last time a lease grab was signaled for the partition (This is required for the case when a node dies somewhere after step 3 but before step 6 has completed). If this happens also clear the lease allocation and lease grab hash for the partition before raising the notification since it is an indication that a node has gone offline without cleaning up.
2. Upon receipt of this notification the timer for this publications is reset (So generally only one publication per partition will be sent during the normal lease time, but it can happen twice if two nodes send them out at the same time. Also when this is received each node will wait based on this formula.
• If the node currently is already processing the partition it will wait the number of active partitions on the node currently held times the current leases held delay minus half of this delay (So basically (Locally active partitions - 1) * current leases held delay).
• If the node currently is not busy processing the partition that is being grabbed the node should wait the local active partitions plus one times the current leases held delay (On so fewer words (Locally active partitions + 0.5) * current leases held delay).
3. Once the delay is done try to set the lease grab hash key for the partition with the conditional transaction parameter of it not being set.
• Generally the node that has the lowest delay from step 2 should get this which also means that the active partitions on each node should distribute evenly among any active nodes since the more active partitions each individual node has the longer it will wait in step 2 and the less likely it is that they will win the race to own the partition lease.
• If a node is currently processing a partition but did not win the race in step 2 it should immediately signal its partition to gracefully shut down and once it is shut down it should remove the lease allocation hash field for the partition. Once this is done it should also publish the allocated lease subscription notification. After that is completed this node should skip the rest of the steps.
4. Check by reading the lease allocation hash value to see if another node than the winner in step 3 is currently busy processing the partition. If this is the case either wait for either the allocated lease subscription notification signaling that the other node has finished from step 3b or if this does not happen wait for a maximum of maximum shutdown time and start the partition anyway.
5. Mark the lease allocation hash with the new current node that is now processing this partition.
6. Also after the minimum lease grab time remove the winning indication in the lease grab hash key for the partition so that it can be challenged again from step 1.

When I run this algorithm in my tests it works exactly as I want it. Once a new node comes online within the normal lease time the workload has been distributed evenly among the new and old nodes. Also an important test is that if you only have one partition the partition does not skip among the nodes, but squarely lands on one node and stays there. And finally if I kill a node without giving it any chance to do any cleanup after roughly maximum lease time the load is distributed out to the remaining nodes.

This algorithm does not in any way handle the case when the load on the different partitions is not uniform, in that case you could relatively easily tweak the formula in step 2 above and replace the locally active partitions with whatever measurement of load or performed work you wish. It will be tricky to keep the algorithm sticky though with these changes.

Fun with collations and persistence layers

So today I spent several hours trying to figure out how to get the correct collation on a SQL Server table created using the Microsoft Entity framework.

So you would think that given that I use a C# string that is UTF8 that when you create a table using this data type with the entity framework it would by default have a collation that supports Unicode more or less properly out of the box... No such luck.

And to make matters worth the Entity framework provides no provision to specify the collection used. In the end what I had to do is to run the following custom SQL on every column that I need proper unicode handling of (For me that means properly distinguishing different high code point Unicode characters as being different letters and also being case sensitive).

  ALTER TABLE {tablename}
    ALTER COLUMN {columnname}
    NVARCHAR({length})
    COLLATE Latin1_General_100_BIN2

The problem is that to do this you also need to drop any indexes that use this column before you make the change and then recreate the column (In my case since I have no deployed database yet I simply create the indexes after I have properly changed the collation). Then once I solved everything for SQL Server I had to do it all over again for MySQL (Different parts of the service runs on SQL Server and MySQL), but that was a lot easier than for SQL Server. Finally I spent some more time trying to figure out some unit tests to validate that an error in this area will not sneak in at a later date since this would be easy to forget.

I have always had a very skeptical approach to persistence layers like the Entity framework (Or indeed any other similar project like Hibernate, XPO etc...) in that I think it produces horribly inefficient database access patterns and should really only be used if you don't care at all about performance or if you are lazy.

I do like certain aspects of the Entity framework though. Specifically I do like being able to specify the database structure using LINQ instead of SQL (You generally get better compile time error checks with this) and also the migration layer for the Entity framework is pretty nice.

That said having worked on developing SQL development tools for over a decade and knowing SQL at the back of my spine I would not trust it for a second to actually generate the SQL for my database access layer. That I hand code for exactly the purposes that I need.

Finally someone explained why Sweden has so much better IT infrastructure than the US

Ran into this article about why Sweden has so much better internet connectivity than the US. I've been complaining about this for years that even now 10 years later I am still paying more for less bandwidth than I had before I moved in Sweden (And this is unfortunately the norm, not a fluke). The reason is quite simple that government here don't dare to tread on the toes of large entrenched economic interests.

I just achieved true Inbox Zero for the first time in over 5 years

For the first time in over 5 years I have achieved true Inbox Zero meaning that I do not have a single message sitting in either my personal or work email inbox. I decided to make a concerted effort to achieve this earlier in the week and after archive over 1500 messages I am done and it feels great!

Removing old inbox messages start getting really easy once you get a couple of months back in your list since very little of that is current any more. It's just that if you never take the time to go through the whole list you never get around to removing this from your inbox and if you have something that is actually important you will have a real hard finding it. In the emails that were over 1 year old I found less than 10 emails in my case that contained information that was still relevant and important. That said I would have a real problem finding that information since it was just sitting around the 1000+ un-filed messages in my inbox.

Now it is just interesting to see how long I can hang on to this.

Why I don't like Windows Phone 8

I attended Microsoft BUILD this weekend and all the attendees got, among other things, a Nokia Lumia 920. I've read a lot and been intrigued by Windows Phone in the past but has never really used it except for a few minutes at a time so I figured I would just throw caution to the wind and move my SIM card over and start using it as my primary phone for as long as I could take it. So here is why I am now after a little bit less than a week switch back to my Galaxy Nexus phone.

The hardware of the phone is excellent, even though the phone is a bit big for my taste (Read thick). But it has good battery life, a truly phenomenal camera and solid build quality. My problem is instead with the software.

First of all I hate that Microsoft has decided that the rules for how to use capital letters in the English language no longer applies to them (And unfortunately this is something that is carried over to Windows 8 and their other new releases too (See the all caps menu bar in Visual Studio 2012 for instance). The page that most epitomizes this on Windows Phone is the setting menu which has the top caption in all caps and then everything else on the entire screen is in lower caps letters except a few random lines that still have capital letters. My head hurts just looking at it.

Another problem with the settings menu is that there is no way (Except through some third party apps) to get a shortcut to certain often used settings, such as WiFi, Bluetooth, brightness etc... They also have completely screwed up the order of the settings. As an example airplane mode is something that to me is pretty used pretty often and on Windows Phone it is not even on the first page when you go into the settings application, but instead you have to scroll down to find it.

There are also a not inconsiderable amount of just plain bugs in it. Yesterday for instance it simply refused to let me answer an incoming call (The answer button was grayed out for some reason).

Next let's look at the search button. The phone has a big nice search button. However this button is not contextual, it is simply a shortcut to Bing, which leads to some really weird UI:s as in for instance the marketplace app where you have two identical search buttons one just above the other. The top one to search the marketplace and the second one for that all important immediate Bing fix.

I also dislike how the back button works, and just think that Android has got this use case pinned down better now (They used to behave similar to Windows Phone). To me you should never be able to go "back" further than the last time you were at the start screen and I just get confused where I will go when I can.

It is also a real shame that you are stuck with the stock keyboard. Granted it is an OK keyboard (Probably about on par with what you get on an iPhone). But if you compare that to Swipe or my personal favorite SwiftKey on Android there is a long way to go unfortunately.

The map application is pretty good and it seems to use a lot less memory to have maps offline than Google Maps does. The turn by turn navigation is definitely not as good though. It doesn't speak street names and as far as I can tell there is only driving directions (No public transportation or walking).

And finally it can't be ignored that the Marketplace (The app store for Windows Phone) is a desolate wasteland compared to Android or iOS. The number of apps does not truly show how bad it is, because even the applications that do exist are usually a lot less functional on Windows Phone than they are on other platforms (The Windows Phone version of Audible is horrible compared to the Android one). Right now the apps that I really need that are not available at all on Windows Phone are Run Keeper, Google+, Picasa (That actually integrates with the phone's photos, not a stand alone app), Google Voice (That integrates with the normal dialer) plus most games that I like to play.

That said there are several things I really like. For instance, the live tile concept is great! Now if only more of the apps used it properly. For instance the client for Picasa that I found didn't use live tiles at all, which seems very odd for a picture app. All, in all the platform itself does show a lot of promise and given a year or two they might be at a point where it is actually useful. Unfortunately it is not yet there.

Americas most epic undertaking started 50 years ago to this day

It is today 50 years ago that John F Kennedy gave his inspiring we choose to go to the moon speech. With this speech started the most epic of scientific achievements probably in the history of mankind to this date.

It is important to remember how the USA was the nation that accomplished this amazing feat as we are slipping in international rankings for math and science educations. It saddens me that we are not still prioritizing this frontier more. For instance we seem poised to spend more than a quarter of the entire NASA budget this year just on the presidential election campaigns. Wouldn't it be better if we had some campaign finance reform to reduce the ridiculous money spent in US politics and use that money more wisely. And in the process we might also end up with politicians that are more interested in the common good than special interest.

I recommend everybody to watch the entire speech below. It is very inspiring and still hugely relevant.

How to steal a design from Apple

Samsung has met their match when it comes to stealing Apple designs. Not only did GooPhone released their new I5 model based on the leaks of the design of the as yet announced iPhone 5. They actually patented the design ahead of Apple and they are threatening to sue Apple if a similar phone is introduced to the Chinese market by them.

Keeping track of your finances

I just thought I would go through some of the sites that I have found that helps you to keep track of your finances that are around on the net. All of this only applies to the US currently. Swedish people can be happy to know that in general we are way ahead of the US in both features, security and usability when it comes to online banking and trading.

The first site I want to mention is Mint. This is a great site that can not only keep track of all the transactions from all of your financial institutions (I have quite a lot of accounts spread out and so far every single one was supported by them) so that you can monitor them for fraudulent transactions easily in one place. It also shows you trends of spending and gives you tips on how you can save money based on your spending. Other features of the site is to help you keep track of bills that need to be paid and it can even help you define and keep a budget. I really can't recommend it enough. And it is all completely free!

If you want some help just keeping track of your bills there is also the site BillQ, but I would really only recommend that over Mint if you feel uncomfortable with giving Mint access to your financial accounts.

Next step of keeping track of your finances is to check and monitor your credit. The first site that is great to keep track of your running credit score is Credit Karma which will both provide you with as many updates to your credit score as you want for free and also provide you with a free credit monitoring service. Another awesome application that also provides a free credit monitoring service is LastPass which I've praised in the past for its main password protection feature.

These services will not provide you with your actual credit report though, just if a change has happened and how it affected your credit score. To get your actual credit report there is an awesome service called Quizzle that gives the ability to get a free credit report every 6 months for free with no credit card required. If you get a credit alert from one of the other services this is also the cheapest premium service for $7 per month you can get as many full credit reports as you want and you can start and end this service easily any time you want. I have cancelled this service myself after just one months of service after I discovered something weird through my credit monitoring services that I wanted to have a better look at without any problems. Can also here let you know to be aware of the common scam where sites will give you a "free" credit report, but you need to provide a credit card and they will usually sign you up to a credit protection service you didn't want (Experian are the experts of this).

Finally if you have investments there are a few sites that are great. When it comes to actual trading I haven't seen anything that comes close to E*Trade, doesn't matter if it relates to actual trading, research, customer support and dealing with taxes it is way better than the other sites I've tried (Except when it comes to fees where Wells Fargo beats them hands down). Doing research and tracking of your investment I use two sites. Yahoo! Finance has a huge amount of statistics and facts available and access to news. Google Finance have slightly less amount of statistics available, but they do have a much nicer interface and also much better real time graphs.

The end of cavities

Scientists have discovered a molecule that when kept in your mouth for 60 seconds will keep your teeth cavity proof for hours. Specifically the molecule kills Streptococcus Mutans and can be added to pretty much any dental product or even candy.

The product still needs FDA approval but once that is hopefully achieved dental care can be changed forever for the better and we will hopefully all be able to keep the teeth we have.

Why we still have spam

According to a report from Microsoft and Google spammers world wide rake in around $200 million in profits, however while doing this they are also costing the receivers of that spam around $20 billion. Unfortunately given that the $20 billion is paid by all of us and not the people that are making the $200 million I don't see them stopping voluntarily anytime soon.

There are fortunately some encouraging developments that could end spam forever (At least in its current incarnation). For instance DMARC which combines Sender Policy Framework and DomainKeys Identified Mail seems a promising technology. If you want to know more about DMARC there is a whole episode on it with Security Now. One of its most important features is that it is supported by all the big guys like AOL, Gmail, Hotmail, Yahoo Mail and Facebook. That said it is something you need to set up on your own mail domain so it will probably take a long time before this is supported universally. Hopefully we are moving in the right direction though.

Why the Apple verdict against Samsung is bad for you

Last week Apple won a lawsuit against Samsung and was awarded over $1 billion in damages. Obviously this is bad for Samsung, but I would argue that it is also bad for all of us, the consumers.

The patents that Samsung was found infringing on were either extremely obvious (Bounce back when scrolling for instance) or just weird (Design patent on a rectangle with rounded edges). It gets even worse when you hear the jurors talk about how they came to the verdict and it is obvious that they have no idea of what they were doing (Which is understandable given that this stuff is very complicated). For instance they decided to skip the discussion about prior art on the patents because "It was bogging us down". Groklaw also has a good rundown on some of the inconsistencies in the jury's verdict.

What will probably happen now is that Android phones will have to jump through a bunch of hoops trying to work around Apple's patents instead of concentrating on adding new awesome features even if Google themselves are trying to down play the significance of the verdict. Also Google has already started to leverage its newly acquired patent portfolio from its acquisition of Motorola so we will see more of this nonsense from all sides. And none of this will get us any better phones because technology is not improved by lawyers, it is improved by engineers.

I would contend that the reason why we have such awesome phones these days is not thanks to either Apple, Samsung or Google. It is thanks to all of them and the fact that they are all trying to put out the absolute best products they can so that they are better than the competition. As Steve Jobs himself was fond of saying "Good artists copy, great artists steal" (In fact even the quote itself is stolen from Pablo Picaso). The copying also goes both ways, tell me the new notifications in iOS wasn't inspired by Android's implementation.

Research and innovation has always been a matter of standing on the shoulders of giants. And most importantly as has happened over and over in the history of science, what happens is that once the body of knowledge gets to a certain point the next step becomes obvious and once anybody thinks about it the next step is usually not that hard and so should not be patentable. I am not saying that there aren't ground breaking leaps of new knowledge that is thought of (Theory of relativity and quantum mechanics stand out), but they are exceedingly rare and for instance inventing a square with rounded corners is not it (In fact if the jury had considered prior art they would have seen that Samsung had prototypes looking like that before the iPhone was released).

Gizmodo does have a different take on this where they are hoping that this will mean the end of the mere copying and the beginning of true innovation. I wish they are right, but I highly doubt it. Also, as they point out, there are quite a lot of innovation already happening in the mobile space. I don't think this will change that either way except that more resources will now be devoted expressly trying to not be similar to the competition instead of trying to make the best possible product.

The next step in human development?

This video discusses how we have already started and will probably continue using technology to improve human performance. We already started using pace makers and prosthethics that might be better than the real thing.

The next logical step is to start enhancing the performance of our brains. We have already began with implants that can treat epilepsy. Is the next step to help people with concentration difficulty really that far off? Where it goes after that is anybody's guess.

The contents of the video is based on the book Amped by Daniel H Wilson.

Did you know that anybody can create new BMW keys?

Did you know that anybody with a kit for around $300 can create their own BMW keys? I didn't and it kind of freaks me out a bit. Especially in a convertible where someone can just jump in (I park with my top down all the time) and get their own key for the car in around 10 seconds.

For some reason BMW has the ODB (On Board Diagnostic) port on the car powered on even though the car is off and through this port you can read the key data needed to program a new key and also add that key to the car. All BMW models are affected that have the option of key less entry except for the new 3-series 2012 sedan, but that could just be because the software hasn't been updated yet.

This is by no mean a BMW only issue, but it is exasperated on for this brand by the fact that this port is on when the car is off and also that it has no security required to access it. BMW is trying to down play the issue, but has as of yet offered no solution.

Finally someone is looking at the high price of a college degree

You might already know about Kahn Academy which provides an interesting and fun way of learning for children up to around a high school level. But what do you do if you are an adult and want to learn a more advanced topic. If you are not the person that is able to get comfortable with a new field by just picking up a book. There is another site aiming for revolutionizing higher level education in the same vein as Kahn Academy is trying to do for lower degrees called Udacity.

Currently the site is free as they are building up their course catalog, but the goal is to eventually offer the equivalent of graduate degrees for as little as $100. A slight difference to what is currently available from universities in the US.

They are also disputing the notion that learning that happens early in your life before you go off and start your career. Instead it thinks that you should keep educating yourself throughout your entire life. Something that I totally agree with.

I completed a couple of classes myself and so far I really like it and can completely recommend it to everybody who has an interest in learning new things. And as I said earlier... It's free!