cloudstack.git
4 days agoAllow KVM VM live migration with ROOT volume on file storage type (#2997) master
Gabriel Beims Bräscher [Fri, 14 Dec 2018 11:01:28 +0000 (09:01 -0200)] 
Allow KVM VM live migration with ROOT volume on file storage type (#2997)

* Allow KVM VM live migration with ROOT volume on file

* Allow KVM VM live migration with ROOT volume on file
- Add JUnit tests

* Address reviewers and change some variable names to ease future
implementation (developers can easily guess the name and use
autocomplete)

11 days agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Fri, 7 Dec 2018 18:16:04 +0000 (23:46 +0530)] 
Merge remote-tracking branch 'origin/4.11'

11 days agotravis: fail fast if --with-marvin fails with nose (#3024)
Rohit Yadav [Fri, 7 Dec 2018 18:15:19 +0000 (23:45 +0530)] 
travis: fail fast if --with-marvin fails with nose (#3024)

* travis: fail fast if --with-marvin fails with nose

Install missing dependency pycrypto.
This fixes issue with recent Travis runs which gave incorrect results
around smoketests with simulator where each test run failed with an
error like "nosetests: error: no such option: --with-marvin".

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
11 days agoserver/test: Move test files to right location (#3085)
Wido den Hollander [Fri, 7 Dec 2018 17:12:52 +0000 (18:12 +0100)] 
server/test: Move test files to right location (#3085)

These files were not in the right directory and thus not being executed
by Maven.

By moving the files we make sure these tests are run again.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
11 days agokvm: Added two more device name patterns to valid bridge slaves (lo* and dummy*)...
Bitworks LLC [Thu, 6 Dec 2018 20:29:00 +0000 (15:29 -0500)] 
kvm: Added two more device name patterns to valid bridge slaves (lo* and dummy*) (#3000)

Added dummy and lo devices to be treated as a normal bridge slave devs.
Fixes #2998
Added two more device names (lo* and dummy*). Implemented tests. Code was refactored.
Improved paths concatenation code from "+" to Paths.get.

12 days agokvm: Use 'ip route show default 0.0.0.0/0' to find the default gateway (#3080)
Wido den Hollander [Thu, 6 Dec 2018 08:34:47 +0000 (09:34 +0100)] 
kvm: Use 'ip route show default 0.0.0.0/0' to find the default gateway (#3080)

If a host has many routes this can be a magnitude faster then printing
all the routes and grepping for the default.

In some situations the host might have a large amount of routes due to
dynamic routing being used like OSPF or BGP.

In addition fix a couple of loglines which were throwing messages on
DEBUG while WARN and ERROR should be used there.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Tue, 4 Dec 2018 11:09:21 +0000 (16:39 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 weeks agoapi: Discover tags field on superclass of API responses (#3005)
Craig Squire [Tue, 4 Dec 2018 08:29:48 +0000 (02:29 -0600)] 
api: Discover tags field on superclass of API responses (#3005)

Updated ApiServiceDiscoveryImpl to check superclasses of API responses for fields.

Fixes: #3002

2 weeks agosecurity: increase keystore setup/import timeout (#3076)
Rohit Yadav [Mon, 3 Dec 2018 19:58:24 +0000 (01:28 +0530)] 
security: increase keystore setup/import timeout (#3076)

This increases and uses a default 15mins timeout for VR scripts and for
KVM agent increases timeout from 60s to 5mins. The timeout can
specifically occur when keystore does not get enough entropy from CPU
and script gets killed due to timeout. This is a very specific corner
case and generally should not happen on baremetal/prod environment, but
sometimes seen in nested/test environments.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 weeks agopackaging: correct permissions in spec file and fix class path specified variable...
Sven Vogel [Fri, 30 Nov 2018 20:08:01 +0000 (21:08 +0100)] 
packaging: correct permissions in spec file and fix class path specified variable (#3030)

Install CentOS 7 e.g. Build 1804 and Java build 1.8.0_181

if you inspect systemd in debug mode you will see some errors
1.
permission of the cloudstack-managment.service are not corretly set
2.
invalid classpath specified. it seems the string which is used will be divided... we now we use ${..} like the lines above ... confused

2 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Fri, 30 Nov 2018 16:56:09 +0000 (22:26 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 weeks agoDestroyvm also removes volumes (#2793)
Henko [Fri, 30 Nov 2018 13:27:31 +0000 (15:27 +0200)] 
Destroyvm also removes volumes (#2793)

* Allow user to detach and delete volumes when destroyinh VMs

* Minor code refactoring

2 weeks agoutils: cleanup Macaddresses utils (#2660)
dahn [Thu, 29 Nov 2018 16:53:50 +0000 (17:53 +0100)] 
utils: cleanup Macaddresses utils (#2660)

Cleanup parse code, fix java docs and remove unwanted comments.

2 weeks agoapi: move ostypeid from DB id to DB uuid, backports #2528 (#3066)
Boris Stoyanov - a.k.a Bobby [Thu, 29 Nov 2018 16:50:51 +0000 (18:50 +0200)] 
api: move ostypeid from DB id to DB uuid, backports #2528 (#3066)

This is a backport to 4.11 of #2528

2 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Thu, 29 Nov 2018 09:36:06 +0000 (15:06 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 weeks agokvm: when untagged vxlan is used, use the default guest/public bridge (#3037)
Rohit Yadav [Wed, 28 Nov 2018 16:52:30 +0000 (22:22 +0530)] 
kvm: when untagged vxlan is used, use the default guest/public bridge (#3037)

When vxlan://untagged is used for public (or guest) network, use the
default public/guest bridge device same as how vlan://untagged works.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 weeks agosg: add secondary ips to the correct ipset based on ip family (#2990)
Rene Diepstraten [Wed, 28 Nov 2018 14:00:13 +0000 (15:00 +0100)] 
sg: add secondary ips to the correct ipset based on ip family (#2990)

Currently secondary ipv6 addresses are added to the ipv4 ipset in security_group.py.
This doesn't work, so this patch adds a function to split a set of ips in ipv4 and ipv6 addresses.
Both the default_network_rules and network_rules_vmSecondaryIp functions now utilise this function and add the ips to the appropriate ipsets.

2 weeks agoFix null pointer exception when using ConfigKey.Scope.ManagementServer (#3062)
Rohit Yadav [Wed, 28 Nov 2018 11:07:45 +0000 (16:37 +0530)] 
Fix null pointer exception when using ConfigKey.Scope.ManagementServer (#3062)

2 weeks agotests: Adding a try catch for Xen licensing issue (#3061)
Boris Stoyanov - a.k.a Bobby [Wed, 28 Nov 2018 11:07:14 +0000 (13:07 +0200)] 
tests: Adding a try catch for Xen licensing issue (#3061)

Adds a skip if the exception is related to licensing of the particular host, for the scaleVM API test.

2 weeks agodb: Add KVM Guest OS mapping for Windows Server 2019 (#3020)
Andrija Panic [Wed, 28 Nov 2018 03:23:32 +0000 (04:23 +0100)] 
db: Add KVM Guest OS mapping for Windows Server 2019 (#3020)

Add KVM Guest OS mapping for Windows Server 2019

3 weeks agomarvin: add missing default test data (#3055)
Rohit Yadav [Tue, 27 Nov 2018 18:47:13 +0000 (00:17 +0530)] 
marvin: add missing default test data (#3055)

This add missing test data for one of the keys for a recently added
migration test.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
3 weeks agoapi: Allow password enabled for iso (#2745) (#3042)
Anurag Awasthi [Tue, 27 Nov 2018 18:39:20 +0000 (00:09 +0530)] 
api: Allow password enabled for iso (#2745) (#3042)

Both register and update iso commands can allow for allowing password
enabled.
Fixes: #2745

Fixe: #2745

3 weeks agoFix null pointer exception when using ConfigKey.Scope.ManagementServer 3062/head
GabrielBrascher [Tue, 27 Nov 2018 12:56:44 +0000 (10:56 -0200)] 
Fix null pointer exception when using ConfigKey.Scope.ManagementServer

3 weeks agoSupport requesting a specific IPv4 address (#2595)
Gabriel Beims Bräscher [Tue, 27 Nov 2018 11:39:10 +0000 (09:39 -0200)] 
Support requesting a specific IPv4 address (#2595)

This commit allows deploying VMs with a specific IPv4 address.

DirectPodBasedNetworkGuru does not support requesting a custom
IP-Address while creating a new NIC/Instance, throwing the following
error:

    Error 530: Does not support custom ip allocation at this time:
NicProfile[0-0-null-null-null
    Unknown macro: { "cserrorcode"}

Some use-cases prefer the ability to request the IPv4 address which the
Instance will get.

This implementation adds unit test cases to cover and it was manually
tested in Basic Networking. I can perform more tests if requested.

3 weeks agoipv6: Advanced Networking Security Groups are supported (#3053)
Wido den Hollander [Sat, 24 Nov 2018 09:40:36 +0000 (10:40 +0100)] 
ipv6: Advanced Networking Security Groups are supported (#3053)

With earlier work in Basic Networking and the security group provider IPv6 is
supported and we can allow IPv6 to be supplied in networks with SG enabled.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
3 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Wed, 21 Nov 2018 07:40:20 +0000 (13:10 +0530)] 
Merge remote-tracking branch 'origin/4.11'

3 weeks agoMerge branch 'origin/4.11.2.0-RC20181113T0924' into 4.11
Rohit Yadav [Wed, 21 Nov 2018 07:38:18 +0000 (13:08 +0530)] 
Merge branch 'origin/4.11.2.0-RC20181113T0924' into 4.11

Doing a ignored RC5 tree merge to get 4.11.2.0 tag on 4.11 branch. This
should have been done before merging the commit to move to
4.11.3.0-SNAPSHOT version on 4.11 branch.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
3 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Wed, 21 Nov 2018 07:30:42 +0000 (13:00 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
3 weeks agoMerge branch 'origin/4.11'
Rohit Yadav [Wed, 21 Nov 2018 07:27:19 +0000 (12:57 +0530)] 
Merge branch 'origin/4.11'

4 weeks agosurefire: ignore system classloader to make tests run (#3038)
Rohit Yadav [Tue, 20 Nov 2018 15:43:44 +0000 (21:13 +0530)] 
surefire: ignore system classloader to make tests run (#3038)

Due to issue described in Surefix bug:
https://issues.apache.org/jira/browse/SUREFIRE-1588

Debian-based users/developers can no longer build CloudStack 4.11+
branches. The other workaround is to have the following jvm property:
jdk.net.URLClassPath.disableClassPathURLCheck=true

Signed-off-by: Rohit Yadav <rohit@apache.org>
4 weeks agoUpdating pom.xml version numbers for release 4.11.3.0-SNAPSHOT
Paul Angus [Tue, 20 Nov 2018 13:11:52 +0000 (13:11 +0000)] 
Updating pom.xml version numbers for release 4.11.3.0-SNAPSHOT

Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
4 weeks agoAdding zone disablement during deletion of the range plus one more tr… (#3028)
Boris Stoyanov - a.k.a Bobby [Mon, 19 Nov 2018 08:28:15 +0000 (10:28 +0200)] 
Adding zone disablement during deletion of the range plus one more tr… (#3028)

## Description
<!--- Describe your changes in detail -->
Disabling the execution of tests that SystemVMs acquire mocked public IPs, since it's braking other tests along the execution.
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->

<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->

## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [X] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [X] Cleanup (Code refactoring and cleanup, that may add test cases)

## Screenshots (if appropriate):
NA
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
The tests were executed and passed locally

<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document -->

4 weeks agoDisplay mac address in nic detail view (#3034)
Philipp Bankonier [Mon, 19 Nov 2018 08:26:03 +0000 (09:26 +0100)] 
Display mac address in nic detail view (#3034)

## Description
Adds the MAC address to the NICs detail view of an instance.

## Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

## Screenshots:

![detailview_with_mac_address](https://user-images.githubusercontent.com/6997263/48567118-a2785d80-e8fc-11e8-9e24-2eda8fcb815a.png)

## How Has This Been Tested?
Tested locally to verify that the correct mac address gets displayed in the detail view.

4 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Wed, 14 Nov 2018 11:23:56 +0000 (16:53 +0530)] 
Merge remote-tracking branch 'origin/4.11'

4 weeks agoserver: remove unused StrategyPriority.PLUGIN. (#3014)
Gabriel Beims Bräscher [Wed, 14 Nov 2018 09:37:37 +0000 (07:37 -0200)] 
server: remove unused StrategyPriority.PLUGIN. (#3014)

Remove unused StrategyPriority.PLUGIN enum. The PLUGIN Strategy priority is not used, except by three JUnit test methods.

4 weeks agotest: Skip network migration tests for not supported hypervisors (#3021)
Nicolas Vazquez [Wed, 14 Nov 2018 09:34:08 +0000 (06:34 -0300)] 
test: Skip network migration tests for not supported hypervisors (#3021)

Skip network migration tests for not supported hypervisors instead of failing.

5 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Tue, 13 Nov 2018 10:43:52 +0000 (16:13 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
5 weeks agoUpdating pom.xml version numbers for release 4.11.2.0 4.11.2.0 4.11.2.0-RC20181113T0924 4.11.2.0
Paul Angus [Tue, 13 Nov 2018 09:24:27 +0000 (09:24 +0000)] 
Updating pom.xml version numbers for release 4.11.2.0

Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
5 weeks agosystemvmtemplate: update debian 9.6 iso url and checksum (#3022)
Rohit Yadav [Tue, 13 Nov 2018 01:00:09 +0000 (06:30 +0530)] 
systemvmtemplate: update debian 9.6 iso url and checksum (#3022)

This fixes the failing systemvmtemplate build due to 404 on old ISO url.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
5 weeks agoCLOUDSTACK-3009: Fix resource calculation CPU, RAM for accounts. (#3012)
Bitworks LLC [Tue, 13 Nov 2018 00:59:08 +0000 (19:59 -0500)] 
CLOUDSTACK-3009: Fix resource calculation CPU, RAM for accounts. (#3012)

The view "service_offering_view" doesn't include removed SOs, as a result when SO is removed, the bug happens. The PR introduces a change for resource calculation changing "service_offering_view" to "service_offering" table which has all service offerings.

Must be fixed in:

4.12
4.11
Fixes: #3009

5 weeks agoMerge pull request #3018 from shapeblue/fixrouterfilecreation
Paul Angus [Mon, 12 Nov 2018 13:25:08 +0000 (13:25 +0000)] 
Merge pull request #3018 from shapeblue/fixrouterfilecreation

Prevent error on GroupAnswers on VR creation

5 weeks agokvm/ceph: When create a new RBD image use modern flags/features for the image (#2985)
Wido den Hollander [Mon, 12 Nov 2018 12:08:54 +0000 (13:08 +0100)] 
kvm/ceph: When create a new RBD image use modern flags/features for the image (#2985)

These additional RBD features allow for faster lookups of how much space a RBD
image is using, but with the exclusive locking we prevent two VMs from writing
to the same RBD image at the same time.

These are the default features used by Ceph for any new RBD image.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
5 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Mon, 12 Nov 2018 06:40:26 +0000 (12:10 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
5 weeks agoagent: add host.reserved.mem.mb parameter documentation (#3016)
Andrija Panic [Mon, 12 Nov 2018 06:10:37 +0000 (07:10 +0100)] 
agent: add host.reserved.mem.mb parameter documentation (#3016)

Document "host.reserved.mem.mb" parameter in agent.properties.

5 weeks agoserver: print log on INFO if Host reached Max Guests Limit (#3013)
Wido den Hollander [Mon, 12 Nov 2018 06:09:17 +0000 (07:09 +0100)] 
server: print log on INFO if Host reached Max Guests Limit (#3013)

This should not be in DEBUG as people would want to know that the host was skipped
because it didn't have enough slots available to run the VM.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
5 weeks agovmware: updateVmwareDc API for updating vmware datacenter details (#2919)
Rohit Yadav [Mon, 12 Nov 2018 06:02:36 +0000 (11:32 +0530)] 
vmware: updateVmwareDc API for updating vmware datacenter details (#2919)

This adds a new API updateVmwareDc that allows admins to update the
VMware datacenter details of a zone. It also recursively updates
the cluster_details for any username/password updates
as well as updates the url detail in cluster_details table and guid
detail in the host_details table with any newly provided vcenter
domain/ip. The update API assumes that there is only one vCenter per
zone. And, since the username/password for each VMware host could be different
than what gets configured for vcenter at zone level, it does not update the
username/password in host_details.

Previously, one has to manually update the db with any new vcenter details for the zone.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
5 weeks agoconfigdrive: Add missing ConfigDrive entries on existing zones after upgrade (#3007)
Nicolas Vazquez [Mon, 12 Nov 2018 06:00:00 +0000 (03:00 -0300)] 
configdrive: Add missing ConfigDrive entries on existing zones after upgrade (#3007)

After upgrade existing environments to 4.11, ConfigDrive cannot be enabled for existing zones due to missing entry on 'physical_network_service_providers' table.

5 weeks agoFix delete domain button in the UI to update only after job finishes (#2905)
Rafael Weingärtner [Sun, 11 Nov 2018 20:41:37 +0000 (18:41 -0200)] 
Fix delete domain button in the UI to update only after job finishes (#2905)

When the “deleteDomain” command is sent via UI, the HTML is already updated assuming that the “deleteDomain” command will actually delete it. However, there are cases (domain with allocated resources) that will not allow ACS to remove the command; as a consequence, the UI will not be in sync with ACS state, which can lead to misunderstandings.

5 weeks agoAdding three new settings into cloud.configuration. (#3019)
Mike Tutkowski [Fri, 9 Nov 2018 22:21:39 +0000 (15:21 -0700)] 
Adding three new settings into cloud.configuration. (#3019)

5 weeks agoPrevent error on GroupAnswers on VR creation 3018/head
nvazquez [Fri, 9 Nov 2018 18:30:57 +0000 (15:30 -0300)] 
Prevent error on GroupAnswers on VR creation

5 weeks agoadd exitcode / distribution detection and remove export CGROUP_DAEMON for RHEL7/CentO...
Sven Vogel [Wed, 7 Nov 2018 20:22:20 +0000 (21:22 +0100)] 
add exitcode / distribution detection and remove export CGROUP_DAEMON for RHEL7/CentOS7 (#2896)

* task/add distribution detection, remove export CGROUP_DAEMON='cpu:/virt'

* taks/add exit values

* change exit codes

* split redhat and centos / change ubuntu to /etc/lsb-release file

* turn around centos and rhel, we need first check for centos then rhel

* set variable distro to None

5 weeks agoForwardmerge #2980 - Fix set initial reservation on public IP ranges
Rafael Weingärtner [Wed, 7 Nov 2018 12:50:09 +0000 (10:50 -0200)] 
Forwardmerge #2980 - Fix set initial reservation on public IP ranges

5 weeks ago[4.11] Fix set initial reservation on public IP ranges (#2980)
Nicolas Vazquez [Wed, 7 Nov 2018 12:48:07 +0000 (09:48 -0300)] 
[4.11] Fix set initial reservation on public IP ranges (#2980)

* Fix initial reservation on public IP ranges

* Do not allow dedicating a system VM IP range

5 weeks agoEnable DPDK support on KVM (#2839)
Nicolas Vazquez [Wed, 7 Nov 2018 12:29:01 +0000 (09:29 -0300)] 
Enable DPDK support on KVM (#2839)

* Enable DPDK support on KVM

* Allow DPDK deployments on user VMs only

* Fix port name ordering

5 weeks agomake sure no time skew can interfere with comparison (#3003)
dahn [Wed, 7 Nov 2018 12:04:40 +0000 (12:04 +0000)] 
make sure no time skew can interfere with comparison (#3003)

* make sure no time skew can interfere with comparison

* import

5 weeks agoForward merge Fix DirectNetworkGuru canHandle for lowercase isolation methods
Rafael Weingärtner [Wed, 7 Nov 2018 11:56:05 +0000 (09:56 -0200)] 
Forward merge Fix DirectNetworkGuru canHandle for lowercase isolation methods

 forward merge of #3010 -- Fix DirectNetworkGuru canHandle checks for lowercase isolation methods

5 weeks agoFix DirectNetworkGuru canHandle checks for lowercase isolation methods (#3010)
Nicolas Vazquez [Wed, 7 Nov 2018 11:53:01 +0000 (08:53 -0300)] 
Fix DirectNetworkGuru canHandle checks for lowercase isolation methods (#3010)

6 weeks agoFix set affinity for user overlay problem (Fixes #2986) (#2987)
Rafael Weingärtner [Thu, 1 Nov 2018 13:30:53 +0000 (10:30 -0300)] 
Fix set affinity for user overlay problem (Fixes #2986) (#2987)

This PR fixes #2986

6 weeks agoapi: signature v3 to accept more formats (#2893)
Yoan Blanc [Wed, 31 Oct 2018 11:27:48 +0000 (12:27 +0100)] 
api: signature v3 to accept more formats (#2893)

It does it by reusing the DateUtil helpers. DateUtil uses java.time.* as that one knows how to deal
with timezones correctly.

The format expected by signatureVersion=3&expires=.... is quite limited.

It should accept the following formats that are containing a timezone and/or milliseconds.

2018-10-01T08:12:14Z
2018-10-01T08:12:14+01:00
2018-10-01T08:12:14+0100
2018-10-01T08:12:14.000Z
2018-10-01T08:12:14.000+01:00
2018-10-01T08:12:14.000+0100
afaik only 2018-10-01T08:12:14+0100 is accepted by the current codebase.

This PR echoes other pull requests I made earlier this year. #2392 and #2867

Signed-off-by: Yoan Blanc <yoan.blanc@exoscale.ch>
7 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Tue, 30 Oct 2018 09:45:35 +0000 (15:15 +0530)] 
Merge remote-tracking branch 'origin/4.11'

7 weeks agokvm: reset KVM host on heartbeat failure (#2984)
Rohit Yadav [Tue, 30 Oct 2018 09:43:59 +0000 (15:13 +0530)] 
kvm: reset KVM host on heartbeat failure (#2984)

On actual testing, I could see that kvmheartbeat.sh script fails on NFS
server failure and stops the agent only. Any HA VMs could be launched
in different hosts, and recovery of NFS server could lead to a state
where a HA enabled VM runs on two hosts and can potentially cause
disk corruptions. In most cases, VM disk corruption will be worse than
VM downtime. I've kept the sleep interval between check/rounds but
reduced it to 10s. The change in behaviour was introduced in #2722.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agodocker: add compose file for database (#2898)
Marc-Aurèle Brothier [Mon, 29 Oct 2018 15:55:48 +0000 (16:55 +0100)] 
docker: add compose file for database (#2898)

A docker-compose file to setup quickly the database for CS. This helps during local development and remove the need to install a local MariaDB/MySQL.

7 weeks agoCLOUDSTACK-9315: Removed unused Classes (#1448)
Gabriel Beims Bräscher [Mon, 29 Oct 2018 11:27:00 +0000 (08:27 -0300)] 
CLOUDSTACK-9315: Removed unused Classes  (#1448)

* Remove some unused Classes

These classes were deleted because they have no references in our code base. They are not in Spring execution flow nor instantiated with "new":
- com.cloud.agent.api.CheckStateAnswer
- com.cloud.agent.api.StartupVMMAgentCommand
- com.cloud.agent.api.routing.UserDataCommand
- remove from description at
com.cloud.configuration.Config.ExecuteInSequenceNetworkElementCommands
enum
- com.cloud.agent.api.storage.UpgradeDiskCommand
- com.cloud.agent.api.storage.CreatePrivateTemplateCommand
- com.cloud.agent.api.storage.DestroyAnswer
- Note: "FIXME: Should have an DestroyAnswer" at
com.cloud.storage.resource.StoragePoolResource
- com.cloud.agent.api.storage.UpgradeDiskAnswer
- com.cloud.agent.api.storage.ManageVolumeAvailabilityAnswer
- com.cloud.agent.api.storage.ManageVolumeAvailabilityCommand
- com.cloud.exception.UsageServerException
- com.cloud.info.SecStorageVmLoadInfo
- com.cloud.serializer.SerializerHelper

* PR#1448 update description of 'execute.in.sequence.network.element.commands' param

Update description of 'execute.in.sequence.network.element.commands'parameter to reflect an unused command that has been removed. The removed class command is 'UserDataCommand'.

* Add cloud schema to update SQL

7 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Mon, 29 Oct 2018 10:57:08 +0000 (16:27 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agovr: defer was broken in VR because of json name change
Rene Diepstraten [Mon, 29 Oct 2018 10:48:25 +0000 (16:18 +0530)] 
vr: defer was broken in VR because of json name change

Committed at f0491d5c72c3161777ca49ae809606a6704df5ff (#2979).

After upgrade from CS 4.10 to CS 4.11, multiple VRs did not start through.
It did not properly defer the finalize config in update_config.py.
Apparently, the json files are now called differently: where it used to
be vm_dhcp_entry.json it now has a uuid added, for example
vm_metadata.json.4d727b6e-2b48-49df-81c3-b8532f3d6745.
The if statement that checks if the finalize can be safely deferred
therefore no longer matches. This PR contains a fix so finalize is
defered again.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agokvm: Fix migrating VM from ISO failures (#2928)
Nicolas Vazquez [Mon, 29 Oct 2018 10:44:20 +0000 (07:44 -0300)] 
kvm: Fix migrating VM from ISO failures (#2928)

Prevents errors while migrating VM from ISO:

Test 1: Deploy VM from ISO -> Live migrate VM to another host -> ERROR
Test 2: Register ISO using Direct Download on KVM -> Deploy VM from ISO -> Live migrate VM to another host -> ERROR

- Prevent NullPointerException migrating VM from ISO
- Prevent mount secondary storage on ISO direct downloads on KVM

7 weeks agovr: defer was broken in VR because of json name change (#2979)
Rohit Yadav [Mon, 29 Oct 2018 10:41:43 +0000 (16:11 +0530)] 
vr: defer was broken in VR because of json name change (#2979)

After upgrade from CS 4.10 to CS 4.11, multiple VRs did not start through.
It did not properly defer the finalize config in update_config.py.
Apparently, the json files are now called differently: where it used to
be vm_dhcp_entry.json it now has a uuid added, for example
vm_metadata.json.4d727b6e-2b48-49df-81c3-b8532f3d6745.
The if statement that checks if the finalize can be safely deferred
therefore no longer matches. This PR contains a fix so finalize is
defered again.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agokvm: use libvirtd as service name for all distributions (#2925)
Rohit Yadav [Mon, 29 Oct 2018 06:25:28 +0000 (11:55 +0530)] 
kvm: use libvirtd as service name for all distributions (#2925)

Since we support only Ubuntu 16.04+ on master/4.12+, we can now use
the libvirt service name `libvirtd` for all distributions. This also
fixes an optional package name for libvirtd installation on Debian 9+.
Fixes #2909

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agoserver: fix unwanted txn commit warning messages (#2927)
Rohit Yadav [Sun, 28 Oct 2018 21:19:54 +0000 (02:49 +0530)] 
server: fix unwanted txn commit warning messages (#2927)

This fixes unwanted transaction commit warning messages such:

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agosaml: redirect saml2 failed login message to a configurable URL (#2185)
Gabriel Beims Bräscher [Sat, 27 Oct 2018 19:25:06 +0000 (16:25 -0300)] 
saml: redirect saml2 failed login message to a configurable URL (#2185)

When a user fails to authenticate with SAML2, it returns an error page
showing the content of the attached image.

To make it more user-friendly and customizable, one could configure a
desirable URL to redirect when such authentication failure happens.

This ticket proposes a global settings variable
(saml2.failed.login.redirect.url). If null, the SAML2 authentication
flow does not change from the current; however, if the user configures
an URL then ACS redirects to that URL.

7 weeks agokvm: improved performance on creating VM (#2923)
alexanderbazhenoff [Thu, 25 Oct 2018 10:58:13 +0000 (13:58 +0300)] 
kvm: improved performance on creating VM  (#2923)

Improved performance on creating VM for KVM virtualization.

On a huge hosts every "ifconfig | grep" takes a lot of time (about 2.5-3 minutes on hosts with 500 machines). For example: ip link show dev $vlanDev > /dev/null is faster than ifconfig |grep -w $vlanDev > /dev/null. But using ip command is much better. Using this patch you can create 500s machine in 10 seconds. You don't need slow ifconfig prints anymore.

7 weeks agonetwork: on rolling restart force stop old routers (#2926)
Rohit Yadav [Thu, 25 Oct 2018 03:50:39 +0000 (09:20 +0530)] 
network: on rolling restart force stop old routers (#2926)

This force stops old VRs when performing rolling restart with
cleanup=true. This will ensure that VRs are powered off quickly than
wait longer for the normal ACPI shutdown. During testing, it was found
on VMware where VM stops are slow compared to XenServer and KVM.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agoMerge remote-tracking branch 'origin/4.11' 2977/head
Rohit Yadav [Thu, 25 Oct 2018 02:44:49 +0000 (08:14 +0530)] 
Merge remote-tracking branch 'origin/4.11'

7 weeks agokvm: HyperV Enlightment for Improved Windows Server 2008+ Performance (#2870)
Simon Weller [Thu, 25 Oct 2018 01:24:13 +0000 (20:24 -0500)] 
kvm: HyperV Enlightment for Improved Windows Server 2008+ Performance (#2870)

Windows has support for several paravirt features that it will use when running on Hyper-V, Microsoft's hypervisor. These features are called enlightenments. Many of the features are similar to paravirt functionality that exists with Linux on KVM (virtio, kvmclock, PV EOI, etc.)

Nowadays QEMU/KVM can also enable support for several Hyper-V enlightenments. When enabled, Windows VMs running on KVM will use many of the same paravirt optimizations they would use when running on Hyper-V.

A number of years ago, a PR was introduced that added a good portion of the code to enable this feature set, but it was never completed. This PR enables the existing features. The previous patch set detailed in #1013 also included the tests.

By selecting Windows PV, the enlightenment additions will be applied to the libvirt configuration. This is support on Windows Server 2008 and beyond, so all currently supported versions of Windows Server.

In our testing, we've seen benchmark improvements of around 20-25% running on Centos 7 hosts and it is also supported on Centos/RHEL 6.5 and later. Testing on Ubuntu would be appreciated.

7 weeks agopackaging: install plugins at /usr/share/cloudstack-management/lib (#2915)
Rohit Yadav [Wed, 24 Oct 2018 12:51:03 +0000 (18:21 +0530)] 
packaging: install plugins at /usr/share/cloudstack-management/lib (#2915)

Install any additional plugin jars in the lib directory to be picked up
by the classpath builder, otherwise one has to manually add the jar
to /etc/default/cloudstack-management after installation. This fixes
the issue for `mysql-ha` plugin.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
7 weeks agotemplate: Return requireshvm key as part of list template response (#2921)
Abhishek [Wed, 24 Oct 2018 10:50:27 +0000 (16:20 +0530)] 
template: Return requireshvm key as part of list template response (#2921)

Added requireshvm key in the response of list template API.
Fixes #2821

Signed-off-by: Abhishek Kumar <abhishek.mrt@gmail.com>
7 weeks agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Wed, 24 Oct 2018 05:38:00 +0000 (11:08 +0530)] 
Merge remote-tracking branch 'origin/4.11'

7 weeks agotemplate: create/updateTemplate should allow to set/change sshKeyEnabled (#2922)
Abhishek [Wed, 24 Oct 2018 05:37:03 +0000 (11:07 +0530)] 
template: create/updateTemplate should allow to set/change sshKeyEnabled (#2922)

Added sshKeyEnabled key in createTemplate and updateTemplate API.
Fixes #2822.

Signed-off-by: Abhishek Kumar <abhishek.mrt@gmail.com>
8 weeks agosystemvm: Ensure cloud service reboots after failure (#2916)
Rohit Yadav [Tue, 23 Oct 2018 18:03:08 +0000 (23:33 +0530)] 
systemvm: Ensure cloud service reboots after failure (#2916)

This fixes an issue for systemvms (CPVM and SSVM) on VMware, as eth0
is not programmed (link-local) the networking.service fails to start
which is a dependency for cloud-postinit service. When cloud-postinit
service fails to start/run, it fails to start the agent (cloud) process.
This fixes the smoketest failures we saw in case of VMware 6.5 with
4.11.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
8 weeks agoForward merge client: mgmt server listen default to 0.0.0.0 (#2907)
Rafael Weingärtner [Mon, 22 Oct 2018 19:28:48 +0000 (16:28 -0300)] 
Forward merge client: mgmt server listen default to 0.0.0.0 (#2907)

8 weeks agotools: Added '-B' flag for mvn current version cmd (#2910) (#2914)
Abhishek [Mon, 22 Oct 2018 16:34:42 +0000 (22:04 +0530)] 
tools: Added '-B' flag for mvn current version cmd (#2910) (#2914)

maven command for finding current version might need to download packages and without batch mode it will end in variable. Added '-B' to enable batch mode.

8 weeks agoclient: mgmt server listen default to 0.0.0.0 (#2907)
Rohit Yadav [Mon, 22 Oct 2018 14:30:51 +0000 (20:00 +0530)] 
client: mgmt server listen default to 0.0.0.0 (#2907)

This makes the management server listen on all interfaces by default.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
8 weeks agokvm: Remove libvirt-bin check for Debian (#2909) (#2912)
Abhishek [Mon, 22 Oct 2018 12:18:11 +0000 (17:48 +0530)] 
kvm: Remove libvirt-bin check for Debian (#2909) (#2912)

Debian does not have libvirt-bin package. Therefore, only for Ubuntu host distro process name libvirt-bin is used, otherwise libvirtd will be used

8 weeks agoserver: Unify templates/ISOs checksum API output (#2911)
Nicolas Vazquez [Sun, 21 Oct 2018 17:03:04 +0000 (14:03 -0300)] 
server: Unify templates/ISOs checksum API output (#2911)

Unify checksum API output for templates and ISOs: not list the checksum algorithm on:
KVM direct downloads

On in progress normal template downloads. The algorithm is shown on the listtemplates API, but after it is downloaded it is not shown anymore.

2 months agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Wed, 17 Oct 2018 14:59:58 +0000 (20:29 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 months agonetwork: Allow ability to disable rolling restart feature (#2900)
Rohit Yadav [Wed, 17 Oct 2018 14:57:08 +0000 (20:27 +0530)] 
network: Allow ability to disable rolling restart feature (#2900)

This adds a global setting for admins who may not want the rolling
restart of routers or are seeing any issues around it. In future, this
setting may be removed.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 months agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Wed, 17 Oct 2018 07:55:06 +0000 (13:25 +0530)] 
Merge remote-tracking branch 'origin/4.11'

2 months agoagent: on shutdown don't allow server reconnection (#2904)
Rohit Yadav [Wed, 17 Oct 2018 01:01:13 +0000 (06:31 +0530)] 
agent: on shutdown don't allow server reconnection (#2904)

When agent is stopped, don't allow reconnection. Previously this would
send a shutdown command to the management server which would put the
host state to Disconnected but then agent's reconnection logic may kick
in sometimes which would connect the agent to the management server
but then the agent process would terminate causing the host to be
put in Alert state (due to ping timeout or it waiting too long).

This fixes the issue by ensuring that when the agent is stopped, it
does not reconnect to the management server.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 months agoAllow enable the Java remote Debug for CloudStack Agent (#2897)
Gabriel Beims Bräscher [Tue, 16 Oct 2018 18:57:13 +0000 (15:57 -0300)] 
Allow enable the Java remote Debug for CloudStack Agent (#2897)

The idea is to create a property in /etc/defaults/cloudstack-agent that
can be enabled/disabled (uncommented/commented) to control the enabling
of Java remote debug in CloudStack's JVM.

2 months agoMerge forward #2902 (Add checksum sanity validation on template registration)
Rafael Weingärtner [Tue, 16 Oct 2018 13:25:50 +0000 (10:25 -0300)] 
Merge forward #2902 (Add checksum sanity validation on template registration)

2 months agoAdd checksum sanity validation on template registration (#2902)
Nicolas Vazquez [Tue, 16 Oct 2018 13:21:20 +0000 (10:21 -0300)] 
Add checksum sanity validation on template registration (#2902)

* Add checksum sanity validation on template registration

* Refactor

* Rename checksum sanity method

2 months agoparams: allow signatureversion and expires without logging
Pierre-Yves Ritschard [Mon, 15 Oct 2018 12:15:00 +0000 (14:15 +0200)] 
params: allow signatureversion and expires without logging

This patch considers the new expires and signatureversion parameters
valid. Without this, all calls log when using the V3 signature scheme.

2 months agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Tue, 16 Oct 2018 05:24:39 +0000 (10:54 +0530)] 
Merge remote-tracking branch 'origin/4.11'

2 months agoagent: set log level to INFO as default for http wire (#2903)
Nicolas Vazquez [Tue, 16 Oct 2018 05:02:03 +0000 (02:02 -0300)] 
agent: set log level to INFO as default for http wire (#2903)

Avoid logging bytes on direct download on KVM.

2 months agovr: memory and swap optimizations (#2892)
Rohit Yadav [Tue, 16 Oct 2018 04:59:48 +0000 (10:29 +0530)] 
vr: memory and swap optimizations (#2892)

This tries to provide a threshold based fix for #2873 where swappinness of VR is not used until last resort. By limiting swappiness unless actually needed, the VR system degradation can be avoided for most cases. The other change is around not starting baremetal-vr by default on all VRs, according to the spec https://cwiki.apache.org/confluence/display/CLOUDSTACK/Baremetal+Advanced+Networking+Support only vmware VRs need to run it and that too only as the last step of the setup/completion, so we don't need to run it all the time.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 months agomerge forward #2876 -- PULL_REQUEST_TEMPLATE simplification
Rafael Weingärtner [Mon, 15 Oct 2018 18:33:13 +0000 (15:33 -0300)] 
merge forward #2876 -- PULL_REQUEST_TEMPLATE simplification

2 months agoPULL_REQUEST_TEMPLATE: simplify and remove unpopular sections (#2876)
Rohit Yadav [Mon, 15 Oct 2018 18:21:13 +0000 (23:51 +0530)] 
PULL_REQUEST_TEMPLATE: simplify and remove unpopular sections (#2876)

This removes the section from the pull request template that is not very
popular or filled by the PR author.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2 months agoRemove push force test file
Rafael Weingärtner [Sat, 13 Oct 2018 12:40:23 +0000 (09:40 -0300)] 
Remove push force test file

2 months agotest force push
Rafael Weingärtner [Fri, 12 Oct 2018 13:58:41 +0000 (10:58 -0300)] 
test force push

2 months agoMerge remote-tracking branch 'origin/4.11'
Rohit Yadav [Wed, 10 Oct 2018 10:40:26 +0000 (16:10 +0530)] 
Merge remote-tracking branch 'origin/4.11'

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>