Taxation issues in Austria, Germany and Czech Republic
Incident Report for Mews
Postmortem

Problem

Due to VAT legislation changes related to COVID-19 in Germany, Austria and Czech Republic, on July 1st, we were performing a migration of relevant accounting items. A portion of accounting items were unfortunately migrated with an incorrect tax rate. As a result of this, our customers had to resolve some of the issues with their settings manually.

Specific issues

  • In Germany, some accounting items was migrated to a tax rate of 50 %, instead of 5 %.
  • All accounting items prior to 1st of July were also migrated if they have been in property’s Editable Accounting History Window.
  • In some cases, tax rates were completely removed from accounting items and property settings.

Actions

  • We established a special project team the moment we uncovered the fact the migration didn’t run successfully. This team was responsible for fixing the related issues in a shortest possible timeframe and communicate with customers affected by this.
  • We started fixing the accounting items using a database backup and a revised version of the migration script.

Causes

  • These changes were announced by relevant governments with an extremely short lead time, leaving us little time to prepare, creating a pressure for all involved parties.
  • Misconfiguration of the migration tool, specifically, by mistake, setting up 50 % tax instead of 5 %.
  • Migrating items inside Editable Accounting History Window is a standard practice when it comes to tax migrations. However, this time, we did not communicate clearly enough the impact of this to our customers. As a result, some of them chose not to close certain accounting items on a bill, hence keeping them open for the migration.
  • When fixing the initial problem, we executed a recovery plan that needed cooperation of our customers, rather than spending more time seeking for a plan that would display issues for a longer time but would allow for a completely unattended recovery.
  • Mews did not offer a functionality that would allow for a batch update of accounting items.

Solutions

  • We have updated our processes with steps, communication templates and new mandatory checks and validations that cover the aforementioned issues. This includes processes for recovery, in case that’s needed.
  • We have added new testing automation that prevents regression of the above issues, where possible.
  • We have implemented mechanisms that make it impossible to set up a tax rate that’s outside of the usual range.
  • We have designed an unattended recovery strategy.
  • We have introduced a product batch update form that allows our customers to update already posted accounting items. We are also looking to introduce other ways for other types of items like rates and nights.
Posted Jul 31, 2020 - 14:02 CEST

Resolved
We're now performing followup actions and working on the postmortem which we will publish soon.
Posted Jul 08, 2020 - 13:27 CEST
Monitoring
We have successfully recovered all the recoverable data that have been corrupted during the incorrect migration. That means all open items that had incorrect VAT rates and all price settings. Therefore it's safe again to start closing bills. For the items that have already been closed on a bill, we did leave them as they are in order not to modify the closed bills. We started engaging directly with all the impacted clients and working on resolution of those closed bills. And we continue to monitor the situation in order to resolve all possible issues that might still arise.

We strongly recommend that you check your pricing setup and future items. If you encounter anything that still does not look correct, let us please know at support@mews.com. We'd like to apologize again for this issue and for the inconvenience this must have caused you. Since this was pretty major incident, we will immediately start working on a detailed postmortem, that will explain what happened and what measures we will put in place so that something similar does not occur again.
Posted Jul 02, 2020 - 23:26 CEST
Update
We've progressed with the recovery, now we are finalizing several remaining items and corner cases that are the most difficult to fix. We'd like to apologize and thank for the patience so far, it's definitely taking more time than it should have. We will update you when the system is fully operational and closing bills becomes safe again.
Posted Jul 02, 2020 - 17:19 CEST
Update
Applicable to hotels in Austria, Germany & Czech Republic ONLY

Unfortunately we had a bug in the migration of the VAT changes, this caused some errors in the VAT setup. We have now successfully reverted the vast majority of the data from a backup and will continue working on several remaining issues to ensure 100% accuracy.
We ask all hotels in Austria, Germany and Czech Republic to manually verify the VAT rates assigned to every Product, Rates and Services in the settings of their hotel/hostel. Once you have verified this, kindly e-mail us at support@mews.com as soon as possible with the outcome of your investigation.

Once we hear from you, we will run another check, to ensure that all open items are then correctly showing the right VAT rates.
Posted Jul 02, 2020 - 10:24 CEST
Update
The recovery is taking longer than expected, but we're moving forward and already fully recovered part of the data. We're continuing to work on the reminder.
Posted Jul 02, 2020 - 04:50 CEST
Identified
We have identified root cause of the issue and are now working on recovery of the corrupted data from our backup.
Posted Jul 02, 2020 - 01:27 CEST
Update
We are continuing to investigate this issue and identified that also Czech Republic is impacted.
Posted Jul 02, 2020 - 00:20 CEST
Investigating
We're investigating an issue related to changes in taxation in Germany and Austria. Until further notice, we recommend not to close any bills until we identify the root cause and resolve the issue. We'll keep you updated here.
Posted Jul 01, 2020 - 23:49 CEST
This incident affected: Operations.