[DOCS] Adds tutorial for getting started with security #68

lcawl · 2018-06-25T23:37:59Z

The Stack Overview contains a tutorial for installing Elasticsearch, Kibana, Logstash, and Metricbeat.

This PR creates a new tutorial that describes how to set up user IDs and passwords in the native realm in that environment.

It does not cover how to encrypt communications, however that can be added in this (or another) tutorial hereafter.

tvernum

This is great step forwards.
I left a few suggestions.

tvernum · 2018-06-26T03:13:52Z

docs/en/stack/security/get-started-security.asciidoc

+major version, you cannot start a new trial. For example, if you have already 
+activated a trial for v6.0, you cannot start a new trial until v7.0.
+
+At the end of the trial period, your license reverts to a Basic license. If you 


Is this correct?
I didn't think we reverted automatically, but I may have missed something.

I don't think it is correct either; the license expires and elasticsearch may operate in a degraded state if security is enabled. That is stats and health apis are not available.

Thanks, I've fixed that here and elsewhere.

tvernum · 2018-06-26T03:20:33Z

docs/en/stack/security/get-started-security.asciidoc

+
+When you use the trial license, {security} is disabled by default. You must 
+stop {kib}, stop {es}, then add the `xpack.security.enabled` setting to the 
+`config/elasticsearch.yml` file in your {es} directory. For example:


This is not really specific to this set of docs, but I've noticed we tend to assume that people reading the docs are using the default zip/tar.gz layout with a config/ directory inside the elasticsearch dir, and not the deb/rpm layout with the config directory in /etc/elasticsearch (or the msi install where you get to pick your location, but at least it defaults to being in the elasticsearch directory).

I don't know what we should do to solve that in general, but maybe here we need to have a note for users of the rpm/deb to check the location of their config directory?

RPM: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/rpm.html#rpm-configuring

DEB: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/deb.html#deb-configuring

I sometimes get caught out on the forums, where my references to the config/ directory leave rpm/deb users very confused.

At one point we used $CONF_DIR in the x-pack docs because of this

I think it's nicer to provide the actual directory, if you can.

Thanks, I hadn't realized it could be in different places. I've added a tip where we talk about the elasticsearch.yml and kibana.yml files. I've also created elastic/elasticsearch#31635 to clean up path issues in some other security pages.

tvernum · 2018-06-26T03:25:58Z

docs/en/stack/security/get-started-security.asciidoc

+Click *Create user* to make a new user. For example, create a user for yourself:
+
+[role="screenshot"]
+image::security/images/create-user.jpg["Creating a user in Kibana"]


Is it worth mentioning that the roles are explained in an upcoming step?
I was surprised that there was no explanation here, until I scrolled down a little.

You're right, I've revamped so we tell them it's coming and then explicitly add the kibana_user role.

tvernum · 2018-06-26T03:28:17Z

docs/en/stack/security/get-started-security.asciidoc

+
+Now go back to the *Management / Security / Users* page and assign these roles 
+to the appropriate users. Assign the `metricbeat_reader` role to your personal 
+user.  Assign the `metricbeat_writer` role to the `metricbeat_internal` user.


I think it's worth mentioning that the personal user should also have kibana_user.

Agreed. If the reader followed your instructions earlier, then this user will already have the kibana_user role, but it's worth mentioning again.

You're right, I've revamped this section to clarify that.

tvernum · 2018-06-26T03:31:31Z

docs/en/stack/security/get-started-security.asciidoc

+Later, when you're ready to increase the number of nodes in your cluster or set 
+up an production environment, you'll want to encrypt communications across the 
+{stack}. To learn how, read <<encrypting-communications>>. 
+


Perhaps mention that the logstash_system and beats_system user are used for monitoring, and refer to those docs?
We told them to remember those passwords when they ran elasticsearch-setup-passwords, but we haven't told them what to do with those users.

Good idea, done!

jaymode · 2018-06-26T20:24:20Z

docs/en/stack/security/get-started-security.asciidoc

+major version, you cannot start a new trial. For example, if you have already 
+activated a trial for v6.0, you cannot start a new trial until v7.0.
+
+At the end of the trial period, your license reverts to a Basic license. If you 


I don't think it is correct either; the license expires and elasticsearch may operate in a degraded state if security is enabled. That is stats and health apis are not available.

jaymode · 2018-06-26T20:25:19Z

docs/en/stack/security/get-started-security.asciidoc

+
+When you use the trial license, {security} is disabled by default. You must 
+stop {kib}, stop {es}, then add the `xpack.security.enabled` setting to the 
+`config/elasticsearch.yml` file in your {es} directory. For example:


At one point we used $CONF_DIR in the x-pack docs because of this

jaymode · 2018-06-26T20:26:30Z

docs/en/stack/security/get-started-security.asciidoc

+----
+
+NOTE: This tutorial involves a single node cluster, but if you had multiple 
+nodes, you would enable {security} on every node in the cluster. If you have a 


maybe you would enable {security} on every node in the cluster and configure Transport Layer Security (TLS) for +internode-communication, which is beyond the scope of this tutorial.

jaymode · 2018-06-26T20:27:39Z

docs/en/stack/security/get-started-security.asciidoc

+The {stack} _authenticates_ users to ensure that they are valid. The 
+authentication process is handled by _realms_. You can use one or more built-in 
+realms, such as the native, file, LDAP, PKI, Active Directory, or SAML realms. 
+Alternatively, you can create your own custom realms. In this tutorial, we'll  


maybe a link to custom realms?

dedemorton

Really great work, Lisa! I ran through the tutorial from end to end (hence all the comments) because I think this is a very important introduction for users. It's going to make it a lot easier for users to get going with security.

dedemorton · 2018-06-27T18:23:52Z

docs/en/stack/security/get-started-security.asciidoc

+
+When you installed the products, they applied Basic licenses with no expiration 
+dates. To view your license in {kib}, go to **Management** and click 
+**License Management**.


Just adding this comment as a reminder that we need to talk to the Kibana team about using a better documentation link. I didn't write down the exact topic link, but the link in the UI drops you into the topic about settings, which seems wrong IMO.

dedemorton · 2018-06-27T18:24:53Z

docs/en/stack/security/get-started-security.asciidoc

+
+When you use the trial license, {security} is disabled by default. You must 
+stop {kib}, stop {es}, then add the `xpack.security.enabled` setting to the 
+`config/elasticsearch.yml` file in your {es} directory. For example:


I think it's nicer to provide the actual directory, if you can.

dedemorton · 2018-06-27T18:27:17Z

docs/en/stack/security/get-started-security.asciidoc

+=== Enable security in {es}
+
+When you use the trial license, {security} is disabled by default. You must 
+stop {kib}, stop {es}, then add the `xpack.security.enabled` setting to the 


I might be assuming the most novice user possible here, but since this is a getting started....

I'd add more info to make it clear that they stop Kibana and ES running in the terminal windows. Otherwise, they might be looking in the Kibana UI for some way to stop everything.

dedemorton · 2018-06-27T18:31:42Z

docs/en/stack/security/get-started-security.asciidoc

+
+When you use the trial license, {security} is disabled by default. You must 
+stop {kib}, stop {es}, then add the `xpack.security.enabled` setting to the 
+`config/elasticsearch.yml` file in your {es} directory. For example:


This might be the first time the user has ever seen a kibana config file. I wonder if we should explain what it's used for? The stack GS might be a better place for this info...not sure, though, because we use the default configs there, and they don't have to modify the configs (except for Beats when they set up LS).

dedemorton · 2018-06-27T18:33:48Z

docs/en/stack/security/get-started-security.asciidoc

+There are built-in users that you can use for specific administrative purposes:
+`elastic`, `kibana`, `logstash_system`, and `beats_system`. 
+
+Before you can use them, you must set their passwords. Restart {es}, then run 


I think it's worth showing the command for starting ES again. You could just modify your lead in text and add the command to the example:

bin/elasticsearch bin/elasticsearch-setup-passwords interactive

dedemorton · 2018-06-27T18:57:55Z

docs/en/stack/security/get-started-security.asciidoc

+the data to {ls} for additional parsing, so no extra settings are required in 
+{metricbeat}. 
+
+<<gs-start-metricbeat,Start {metricbeat}>>. 


Not sure it's a good idea to send people out of this flow to another topic for a single command. They might lose their place in the docs. I'd just show the command again here.

dedemorton · 2018-06-27T19:05:15Z

docs/en/stack/security/get-started-security.asciidoc

+Log in to {kib} with the user ID that has `metricbeat_reader` role (for example, 
+`jdoe`). 
+
+This user should have authority to visualize the system metrics (for example, on 


I'd change the wording here because readers might wonder whether there's an additional step required. If they've followed the steps in the tutorial, the user will have the authority to visualize system metrics, right?

dedemorton · 2018-06-27T19:05:40Z

docs/en/stack/security/get-started-security.asciidoc

+
+[float]
+[[gs-security-nextsteps]]
+==== What's next?


Shouldn't this be a level === heading?

dedemorton · 2018-06-27T19:09:02Z

docs/en/stack/security/get-started-security.asciidoc

+    hosts => "localhost:9200"
+    manage_template => false
+    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
+    user => metricbeat_internal <1>


I'd put double quotes around the username and password: user => "metricbeat_internal" and password => your_password

The last time I checked, if the password contains dashes, it won't work without the double quotes (see elastic/logstash#9618). I think this is true for user, too, but I haven't tested it.

Also (this comment applies to other sections, too), I wonder if this tutorial should be showing how to use the secrets keystores (in ES and LS), too, so that users don't have to hard code values in their configs. I know this makes the tutorial more complicated, but most users do not want to have passwords revealed in config files, so it's a very common use case.

Good point @dedemorton . I've added steps related to the Kibana and Logstash keystores.

dedemorton · 2018-06-27T19:11:13Z

docs/en/stack/security/get-started-security.asciidoc

+<1> Specify the `metricbeat_internal` user that you created earlier in this tutorial. 
+<2> Specify the password that you chose for this user ID.
+
+<<gs-start-logstash,Start {ls}>>. 


Same comment as before. I think it's better to repeat the command for ease-of-use.

Also, you might want to mention that it might take a few minutes for these changes to be applied. I think I had to restart Logstash twice, but maybe just because I am impatient.

dedemorton

Looks great! Just a few comments. I like having the keystore info, but I think it might help to set it off somehow.

dedemorton · 2018-07-07T00:12:45Z

docs/en/stack/security/get-started-security.asciidoc

+command then save your changes to the file. 
+--
+
+. If you prefer not to put your user ID and password in the `kibana.yml` file, 


1 and 2 are mutually exclusive so I would not make them separate steps. Noticed this because I performed step 1 and then saw step 2...and thought...no, I want to do that instead!

dedemorton · 2018-07-07T00:20:38Z

docs/en/stack/security/get-started-security.asciidoc

+to the appropriate users. Assign the `metricbeat_reader` role to your personal 
+user.  Assign the `metricbeat_writer` role to the `logstash_internal` user.
+
+The list of users should now contain all of the built-in users as well as the 


I really like that you include a screen capture here. It's not good Kibana lets you change tabs without prompting you to Save. You can lose stuff.

dedemorton · 2018-07-07T00:40:05Z

docs/en/stack/security/get-started-security.asciidoc

+    }
+  }
+}
+


If possible, I suggest having a shared file that contains the input and filter parts shown here. I worry that we'll update the example in one topic and forget to update it in another. Part of me wonders if it's even necessary to show the full config.

Good point. I'll omit everything but the elasticsearch section

dedemorton · 2018-07-07T00:48:42Z

docs/en/stack/security/get-started-security.asciidoc

+<1> Specify the `logstash_internal` user that you created earlier in this tutorial. 
+<2> Specify the password that you chose for this user ID.
+
+If you prefer not to put your user ID and password in the configuration file, 


I really went down a rabbit hole because I copy/pasted your first config example and then decided that I wanted to use the keystore instead. So I modified my config, but forgot to add the $ (even though I know it's required). I wasted time recreating the keystore (I thought that I might have made a typo...there's no way to know for sure because when you add a key to the keystore, it doesn't let you confirm the value) .

I might have avoided this particular rabbit hole if I'd gone directly to the keystore config example first and copied that.

You might want to reformat this content or add something to intro so that careless readers (like me!) will know up front that they can use a keystore. Might be worth even pointing out the dollar sign. This was frustrating because it was one of those dumb mistakes that your eyes skip over and the error message is not useful.

The general theme here is that I do no read carefully...but I don't think I'm unique that way.

lcawl · 2018-07-09T17:33:11Z

Thanks for the feedback, @dedemorton ! I've put the config file and keystore options as bulleted options so hopefully it's clearer now that you do one or the other.

dedemorton

LGTM.

lcawl · 2018-07-24T20:36:56Z

I've decided to split this tutorial into multiple pages (i.e. more like the machine learning tutorial).

@tvernum I also added a tip recommending that users explicitly set the xpack.security.enabled setting

lcawl · 2018-07-27T17:03:12Z

Backported to 6.x, 6.4, and 6.3 branches via 335e778

lcawl added v7.0.0 v6.4.0 v6.3.1 labels Jun 25, 2018

lcawl requested review from dedemorton, jaymode and karenzone June 25, 2018 23:38

tvernum reviewed Jun 26, 2018

View reviewed changes

jaymode reviewed Jun 26, 2018

View reviewed changes

lcawl mentioned this pull request Jun 27, 2018

ML getting started docs elastic/elasticsearch#31541

Closed

dedemorton reviewed Jun 27, 2018

View reviewed changes

lcawl added the WIP Work in progress label Jun 27, 2018

lcawl force-pushed the lcawley-gs-security branch from 66d1bec to 88b4743 Compare July 3, 2018 18:51

lcawl removed the WIP Work in progress label Jul 4, 2018

lcawl mentioned this pull request Jul 5, 2018

[DOCS] Updates the introduction in the machine learning tutorial #74

Merged

lcawl added 16 commits July 6, 2018 13:26

[DOCS] Creates security getting started tutorial

a1e4f2a

[DOCS] Adds built-in users to security tutorial

43052c2

[DOCS] Adds user and role steps to security tutorial

7a8c970

[DOCS] Adds metricbeat and logstash config

9a093fa

[DOCS] Small edits and image fixes

57b669f

[DOCS] Fixed config file paths

9e15954

[DOCS] Improved role assignment info

8646df2

[DOCS] Fixed licensing info

707f6e7

[DOCS] Clarify built-in users

3f37d87

[DOCS] Clarified TLS note

f8afc67

[DOCS] Adds examples for start and stop commands

8cfcfe3

[DOCS] Adds quotes to metricbeat config

99dee45

[DOCS] Adds kibana keystore commands

cb8cd59

[DOCS] Adds logstash keystore info

ccacb3c

[DOCS] Addressed final review comments

2d9445e

[DOCS] Changes metricbeat_internal to logstash_internal

9c60bac

[DOCS] Rebase from master

802aa77

lcawl force-pushed the lcawley-gs-security branch from 529ecde to 802aa77 Compare July 6, 2018 20:32

lcawl added 2 commits July 6, 2018 15:37

[DOCS] Remove duplicate image

3f1114a

[DOCS] Split off reusable content

9e29ac3

dedemorton reviewed Jul 7, 2018

View reviewed changes

[DOCS] Clarified keystore vs config file options

fd326a6

dedemorton approved these changes Jul 9, 2018

View reviewed changes

[DOCS] Re-use Kibana configuration info

0dc87b1

lcawl mentioned this pull request Jul 24, 2018

[DOCS] Adds tutorial for getting started with TLS #87

Closed

lcawl added 2 commits July 24, 2018 12:57

[DOCS] Adds xpack.security.enabled recommendation

446e090

[DOCS] Split security tutorial into multiple pages

dd7d54b

lcawl merged commit 6a5fbdb into elastic:master Jul 27, 2018

lcawl deleted the lcawley-gs-security branch July 27, 2018 16:29

lcawl added v6.3.3 v6.5.0 and removed v6.3.1 labels Jul 27, 2018

[DOCS] Adds tutorial for getting started with security #68

[DOCS] Adds tutorial for getting started with security #68

Uh oh!

Conversation

lcawl commented Jun 25, 2018

Uh oh!

tvernum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dedemorton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dedemorton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

dedemorton Jul 7, 2018 •

edited

Loading

lcawl commented Jul 9, 2018 •

edited

Loading

lcawl commented Jul 24, 2018 •

edited

Loading