Sync support #282

andrey-yantsen · 2018-08-19T23:04:27Z

Hi there! I've started working on Mobile Sync support. If anybody has any comments on the code, or just any feature requests — feel free to tell :)

At the beginning the code can do few useful things:

Initialise the "sync" env: tell the plex.tv that the client supports syncing, and the client can be used as sync-target
Unfortunately I have to "fake" used platform/device info, because transcoding is depends on this values
And finally you can download converted files, by iterating over media parts and looking for ones with required syncItemId

The reason why I'm started this is simple: I have a WD My Passport Pro which has the Plex onboard and I'd like to have my "travel" library in sync with the main one. I've already started the project for this (andrey-yantsen/plexiglas) but for now there is nothing to see there.

fix #134

Hellowlol · 2018-08-20T06:59:46Z

Hi! Thanks for the PR!

How do you actually sync the items? Can you add a test?

I would expect that there was a sync function inside the client so you could just pass a list of objects and that would be it.

Any info needed for the transcode can be picked up there this way we don’t need to mess with the init method.

andrey-yantsen · 2018-08-20T07:27:11Z

Just to clarify: the PR is not finished now, it's an early alpha version :)

How do you actually sync the items?

The list of "items-to-sync" is controlled in Plex Web, you can add any item to the list for a specific device.

In the following code block I will show the way to download all suitable content (actually I've not yet reached the "downloading" stage in my project, so I'm not sure if this really working now):

from plexapi import sync, utils
from plexapi.myplex import MyPlexAccount
sync.init()
plex = MyPlexAccount()

for item in plex.syncItems().items:
    if item.status.state != 'complete':
        continue

    for media in item.getMedia():
        for part in media.iterParts():
            filename = '%s.%s' % (item._prettyfilename(), part.container)
            url = item._server.url(part.key)
            filepath = utils.download(url, token=plex.authenticationToken, filename=filename, savepath=os.getcwd(),
                                      session=item.server()._session, showstatus=True)

Can you add a test?

Yep, of course, on some latter stage :)

I would expect that there was a sync function inside the client so you could just pass a list of objects and that would be it.
Any info needed for the transcode can be picked up there this way we don’t need to mess with the init method.

As far as I can see the only way for Plex to detect transcoding options, is thru target platform. Here is what I see in the Plex logs with real platform info:

ERROR - Unable to find client profile for device; platform=Darwin, platformVersion=17.7.0, device=
Darwin, model=

And the process goes smoothly when I mimic my iPhone SE.

Also here is the request which should be sent to add an item to sync list:

POST https://plex.tv/devices/779755E8-54E2-4D0F-A7D9-E2FF8171CC55/sync_items?X-Plex-Device-Name=Yantsen%20-%20iPhone%20An HTTP/1.1

Content-Type:	application/x-www-form-urlencoded

SyncItem[Location][uri]:        library://ac0a6a57-522b-4dcf-9cb8-0a58b16c6726/item/%2Flibrary%2Fmetadata%2F2428
SyncItem[MediaSettings][audioBoost]:100
SyncItem[MediaSettings][maxVideoBitrate]:4000
SyncItem[MediaSettings][musicBitrate]:192
SyncItem[MediaSettings][photoQuality]:74
SyncItem[MediaSettings][photoResolution]:1920x1080
SyncItem[MediaSettings][subtitleSize]:100
SyncItem[MediaSettings][videoQuality]:100
SyncItem[MediaSettings][videoResolution]:1280x720
SyncItem[Policy][scope]:        all
SyncItem[Policy][unwatched]:    0
SyncItem[Policy][value]:        0
SyncItem[contentType]:          video
SyncItem[machineIdentifier]:    607c8c938b50eef734456f8b9da94b5d02339ce5
SyncItem[metadataType]:         movie
SyncItem[rootTitle]:            Test
SyncItem[title]:                Test1

As you can see there are nothing here about container and/or supported codecs list, only basic/common transcoding stuff.

Hellowlol · 2018-08-20T08:05:06Z

You dont really need to do anything to make the sync happens. Seems you just upload zhe shit to a queue, and the client handles the rest from there.

Anyway to send a sync command should be as easy as:

inside client.py


def sync_item(self, item): 
    #magic happens here.
    # Maybe grab stuff that you need from the client like platform etc
    # name, title from item etc.

Looking forward to test the pr :)

andrey-yantsen · 2018-08-20T08:21:10Z

Seems you just upload zhe shit to a queue, and the client handles the rest from there.

Yep, adding new sync-item to some other client is easy, and you don't have to call sync.init() for it. And it's not supported by this PR now :)

My goal is to support "sync-target": when you run the script on some random computer, select sync-items in your Plex Web and the required media magically being copied to this computer, in the suitable format.

In my case I have an external HDD and I'd like to always have some movies / tv shows (converted to mp4 and not in 4k format), e.g. to be able to watch something while I'm on a plane.

Hellowlol · 2018-08-20T08:25:18Z

I dont think syncing to computers is supported by plex at all. Do you just want to copy things from another shared plex server to your own? Or sync to a actual client?

Anyway if you just want to download stuff or transcode to a computer thats already supported.

andrey-yantsen · 2018-08-20T08:28:52Z

I dont think syncing to computers is supported by plex at all.

It's not supported out-of-the-box, but we're the engineers, and we can force software to do what we want! :)

Do you just want to copy things from another shared plex server to your own?

I want to copy optimised copies of things from any plex server to an external hdd

Hellowlol · 2018-08-20T08:38:44Z

Oh, thats easy. Its pretty much supported out of the box.

See the docs for.

https://python-plexapi.readthedocs.io/en/latest/modules/video.html?highlight=getStreamURL%20#plexapi.video.Movie.download

note the comment about **kwargs – Additional options passed into getStreamURL().
You can use this to transcode.

here was a example on how i use it from one my my own projects.

https://github.com/Hellowlol/plex-cli/blob/master/plexcli/cli.py#L126

andrey-yantsen · 2018-08-20T08:45:04Z

Yep, I saw this option, I plan to support it in my project, but it's not on my priority list...

getSteamURL() not that powerful, easy and cute like Mobile Sync: e.g. with Sync you can mark to sync "3 unwatched episodes", "3 unwatched movies with highest rating", "3 unwatched movies released recently", and so on. And all those media would be automatically rotated while you marking something as watched and when new releases become available.

Hellowlol · 2018-08-20T08:53:37Z

I think im missing something here. Do you expect to do this from the plex ui?
If would be easy to get movies/episodes based on some condition run them in some cron script.

andrey-yantsen · 2018-08-20T09:13:24Z

Do you expect to do this from the plex ui?

You're exactly right :) https://support.plex.tv/articles/201082477-quick-guide-to-mobile-sync/

If would be easy to get movies/episodes based on some condition run them in some cron script.

Man... I just like the Mobile Sync approach, where you don't have to code the filters by yourself, but you can easily choose the required things with UI and this UI doesn't require any dev-knowledge at all, even my wife would be able to use it! :D

Hellowlol · 2018-08-20T09:36:23Z

Oh, i think i get it now. You faking so plexapi is detected as a client in the plex webui... imo that should be done with your own project.

Nice idea btw!

andrey-yantsen · 2018-08-20T10:19:49Z

imo that should be done with your own project.

You mean sync.init()? You're right, I'll replace it with a note in the docs.

…ster

When you have multiple media within one SyncItem it takes a lot of time to get all the info for this media (on my machine it takes about a second for each movie).

ghost · 2018-08-24T15:32:58Z

plexapi/base.py

            on how this is used.
        """
-        data = self._server.query(ekey)
+        timeout = kwargs.pop('timeout', None)


This should probably be added as a real keyword arg timeout=None in the function definition and not pulled out of kwargs.

ghost · 2018-08-24T15:34:56Z

plexapi/myplex.py



-def _connect(cls, url, token, timeout, results, i):
+def _connect(cls, url, token, timeout, fast, results, i, connected_event=None):


If X_PLEX_ENABLE_FAST_CONNECT is a global configuration option. Should we instead always use fast=True or false based on that, rather than passing it in as an argument?

ghost · 2018-08-24T15:36:50Z

plexapi/utils.py

-    for thread in threads:
-        thread.join()
-    return results
+    while not connected_event.is_set():


Just curious, what was the issue that led to this change?

Most times I run the my things outside my home network, when the local IP of the server is not available, and it takes annoyingly long time to establish the connection

Makes sense, thanks.

andrey-yantsen · 2018-08-24T15:44:42Z

@mikes-nasuni thank you for the fast reaction, I've made the changes which you've suggested :)

Hellowlol · 2018-08-24T16:04:16Z

plexapi/utils.py

+        if all([not t.is_alive() for t in threads]):
+            break
+        time.sleep(0.05)
+    return list(filter(lambda r: r is not None, results))


Use a list comprehension

Hellowlol · 2018-08-24T16:06:12Z

plexapi/sync.py

-        url = '/sync/%s/%s/files/%s/downloaded' % (
-            self._device.clientIdentifier, server.machineIdentifier, sync_id)
-        server.query(url, method=requests.put)
+    def markDownloaded(self, media: Playable):


This need to be support python 2.

Hellowlol · 2018-08-24T16:07:49Z

plexapi/sync.py

    def server(self):
-        server = list(filter(lambda x: x.machineIdentifier == self.machineIdentifier, self._servers))
+        server = list(filter(lambda x: x.clientIdentifier == self.machineIdentifier, self._server.resources()))
        if 0 == len(server):


List comp please

Hellowlol · 2018-08-24T16:09:06Z

plexapi/sync.py

        return server[0]

-    def getMedia(self):
+    def getMedia(self, timeout=None):


What up with the timeout? Why not just change the plex.Timeout?

Not all operations require 60s+ timeout, and wrapping getMedia() calls with changing timeout back-and-forth would look strange.
You think that just globally change TIMEOUT in my code would be more than enough and wouldn't cause any strange problems?

You think that just globally change TIMEOUT in my code would be more than enough and wouldn't cause any strange problems?

It shouldn't cause any problems. timeout is essentially just telling the requests module when to give up if not getting a response. It's really only useful to change to poor or slow network issues. Having different timeouts for different calls in to the API seems incorrect unless you have a really strong case for it.

Perhaps a better option for your case is to update your personal config file which is not checked in.

Hellowlol · 2018-09-03T18:29:29Z

plexapi/myplex.py

+        """ Returns an instance of :class:`~plexapi.sync.SyncItems` for specified client
+
+            Arguments:
+                 clientId (str): an identifier of a client for which you need to get SyncItems. Would be set to current


allow for both a client and a clientid as a string.

Hellowlol · 2018-09-03T18:30:53Z

plexapi/myplex.py

+        """ Adds specified sync item for the client
+
+            Arguments:
+                 client (:class:`~plexapi.myplex.MyPlexDevice`): pass


doc what this client, sync_item is used for.

Hellowlol · 2018-09-03T18:32:06Z

plexapi/server.py

        return '%s%s' % (self._baseurl, key)

+    def refreshSynclist(self):
+        return self.query('/sync/refreshSynclists', self._session.put)


add missing doc string

Hellowlol · 2018-09-03T18:32:23Z

plexapi/server.py

+        return self.query('/sync/refreshSynclists', self._session.put)
+
+    def refreshContent(self):
+        return self.query('/sync/refreshContent', self._session.put)


add missing doc string

Hellowlol · 2018-09-03T18:33:59Z

plexapi/sync.py

+        status (:class:`~plexapi.sync.Status`): current status of the sync
+        mediaSettings (:class:`~plexapi.sync.MediaSettings`): media transcoding settings used for the item
+        policy (:class:`~plexapi.sync.Policy`): the policy of which media to sync
+        location (str): unknown


self.clientidentifier is not listed as a attribute.

Hellowlol · 2018-09-03T18:34:48Z

plexapi/sync.py

+        """ Returns :class:`~plexapi.myplex.MyPlexResource` with server of current item.
+        """
+        server = [s for s in self._server.resources() if s.clientIdentifier == self.machineIdentifier]
        if 0 == len(server):


can you do len(server) == 0 instead,

Hellowlol · 2018-09-03T18:35:12Z

plexapi/sync.py

+        media._server.query(url, method=requests.put)
+
+    def delete(self):
+        url = SyncList.key.format(clientId=self.clientIdentifier)


Missing doc string

Hellowlol · 2018-09-03T18:36:19Z

plexapi/sync.py

+        self.clientId = data.attrib.get('clientIdentifier')
+        self.items = []
+
+        for elem in data:


Can we use .iter('SyncItem') here?

Hellowlol · 2018-09-03T18:39:05Z

plexapi/video.py

+        return self._prettyfilename()
+
+    def sync(self, client, policy, media_settings, title=None):
+        """ Add current video as sync item for specified device.


add docs for args, and keywork arguments.

Hellowlol · 2018-09-03T18:40:07Z

tests/test_sync.py

+MAX_RETRIES = 3
+
+
+def get_new_synclist(account, client_id, old_sync_items=[]):


old_sync_items is mutable, is this intended?

I've already rewrote this part locally, anyway :D

andrey-yantsen · 2018-09-03T23:58:19Z

Heyhey, @Hellowlol, thank you for the review! I've pushed another pile of code, where I've done few things:

Fixed part of your review request
Marked all (I hope that I marked them all) as TODO, to not miss it later
Simplified initialisation of sync.MediaSettings in order to get this working easily with other content types
Added sync() method to audio, photo & playlists
Each sync-related method now can accept client as an object, or clientId as a string
Covered this stuff with creepy tests

In order to get sync/tests for photos working PR #292 need to be applied.

Hellowlol · 2018-09-04T12:37:33Z

Let me know when this is ready, i want to squash and merge this before i do the small prs.

andrey-yantsen · 2018-09-04T12:38:52Z

Yep, sure. I hope to finish this by tomorrow.

andrey-yantsen · 2018-09-04T20:22:25Z

@Hellowlol here goes the docs :) Anything else you'd like to be changed?

Hellowlol · 2018-09-06T10:59:46Z

Thanks for the doc changes. I havnt really looked at the code before now. There is a lot of repetitions regarding the sync methods. Anyway we could simplify this?

andrey-yantsen · 2018-09-06T11:12:10Z

Uh... From the top of my head — I can add a mixin Syncable with some basic implementation of the method, but, IMO, because of different content types (video, audio, photo) require different options, and it's possible to sync so many different, not directly related in OOP to each other, stuff (library, playlist, movie, tv-show episode, etc.) there are still be some repetition...

@Hellowlol Do you have any better ideas?

Hellowlol · 2018-09-06T11:16:59Z

Using a mixin was my thought too. If there are too many options we might just leave it as it is. Anyway are you done?

andrey-yantsen · 2018-09-06T11:18:42Z

Yep, it's done I think. Probably I'll find something else later, but for now I have nothing to add :)

Hellowlol · 2018-09-06T12:16:05Z

Awesome, thanks!

Hellowlol · 2018-09-06T12:41:58Z

Can you fix the inline imports so the style is consistent? We don't seem to use any other relative imports,

andrey-yantsen · 2018-09-06T12:58:19Z

Yep, sure, I've just pushed it

[sync] initial commit

5432876

andrey-yantsen changed the title ~~[WIP] Sync support~~ [WIP] [Do not merge] Sync support Aug 20, 2018

fix populating of state field in sync.Status

edfe9d5

andrey-yantsen added 6 commits August 24, 2018 16:15

[connection] add posibliity to return first established connection fa…

05bd225

…ster

[base] add timeout argument to PlexObject.fetchItems()

3f9673c

[sync] add timeout arg to SyncItem.getMedia()

1fa3dc4

When you have multiple media within one SyncItem it takes a lot of time to get all the info for this media (on my machine it takes about a second for each movie).

[sync] fix marking media as downloaded

6051a8a

[sync] pass clientIdentifier to created SyncItem()

5ec0a42

[sync] override __repr__() for sync.Status

1f11bdf

ghost reviewed Aug 24, 2018

View reviewed changes

fix after @mikes-nasuni`s review

cd2849e

Hellowlol reviewed Aug 24, 2018

View reviewed changes

andrey-yantsen added 2 commits August 24, 2018 17:18

fix python2 compatibility

afbf45c

get rid of sync.init()

34296c7

Hellowlol reviewed Sep 3, 2018

View reviewed changes

sync for everybody!

9971435

andrey-yantsen added 3 commits September 4, 2018 01:03

add TODO doctring for Audio._defaultSyncTitle()

5a147f8

SyncItems tag may be presented only once, there is no need for loop

98d93d3

add more TODO docstrings

24c974f

hello docs

e82e7d7

andrey-yantsen changed the title ~~[WIP] [Do not merge] Sync support~~ Sync support Sep 4, 2018

andrey-yantsen added 2 commits September 6, 2018 13:54

remove relative import

ee867f5

remove unused variable from tests/test_sync.py

462a16d

Hellowlol merged commit 54b26fd into pushingkarmaorg:master Sep 8, 2018



		def _connect(cls, url, token, timeout, results, i):
		def _connect(cls, url, token, timeout, fast, results, i, connected_event=None):

		MAX_RETRIES = 3


		def get_new_synclist(account, client_id, old_sync_items=[]):

Sync support #282

Sync support #282

Uh oh!

Conversation

andrey-yantsen commented Aug 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hellowlol commented Aug 20, 2018

Uh oh!

andrey-yantsen commented Aug 20, 2018

Uh oh!

Hellowlol commented Aug 20, 2018

Uh oh!

andrey-yantsen commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hellowlol commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrey-yantsen commented Aug 20, 2018

Uh oh!

Hellowlol commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrey-yantsen commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hellowlol commented Aug 20, 2018

Uh oh!

andrey-yantsen commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hellowlol commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrey-yantsen commented Aug 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrey-yantsen commented Aug 24, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Aug 24, 2018 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

andrey-yantsen commented Aug 19, 2018 •

edited

Loading

andrey-yantsen commented Aug 20, 2018 •

edited

Loading

Hellowlol commented Aug 20, 2018 •

edited

Loading

Hellowlol commented Aug 20, 2018 •

edited

Loading

andrey-yantsen commented Aug 20, 2018 •

edited

Loading

andrey-yantsen commented Aug 20, 2018 •

edited

Loading

Hellowlol commented Aug 20, 2018 •

edited

Loading

ghost Aug 24, 2018 •

edited by ghost

Loading

Hellowlol commented Sep 6, 2018 •

edited

Loading