Fix issues related to keyboard inset #37719

vashworth · 2022-11-17T18:21:55Z

Refactors keyboard logic to use a combination of UIKeyboardWillShowNotification, UIKeyboardWillChangeFrameNotification, UIKeyboardWillHideNotification notifications to determine the keyboard inset.

When a docked keyboard is shown, or when it changes from undocked/floating to docked, UIKeyboardWillShowNotification is triggered. And when a keyboard is hidden or changes from docked to undocked/floating, UIKeyboardWillHideNotification is triggered. So it would make sense to calculate and set the inset when UIKeyboardWillShowNotification happens and set the inset to 0 when UIKeyboardWillHideNotification happens.
However, there are some cases where these events are missing, have incomplete values, or do something unexpected, so we use UIKeyboardWillChangeFrameNotification in addition to help handle those cases.

Note: This fix makes a best effort to accommodate Stage Manger, but does not fix all issues related to Stage Manager. In addition, it does not address issues related to the inset when the keyboard is open and the screen orientation is rotated.

Fixes flutter/flutter#86847.
Fixes flutter/flutter#104689.
Fixes flutter/flutter#109845.
Fixes flutter/flutter#115528.
Fixes flutter/flutter#53565.

Issue	Before	After
flutter/flutter#86847
flutter/flutter#104689
flutter/flutter#109845
flutter/flutter#115528, flutter/flutter#53565
flutter/flutter#115528, flutter/flutter#53565

Other use cases I tested manually. I tested each case in Full Screen View, Split View, Slide Over View. I tested each case with a custom keyboard with a floating decimal height (see Custom keyboard extension code sample below).

Docked Keyboard
- Opening/Closing
- Dragging docked keyboard and then putting back
- Rotating orientation while keyboard open
Undocked Keyboard (iPad only)
- Changing from docked to undocked via menu
- Opening/Closing
- Dragging to a new position
- Changing from undocked to docked via menu
- Dragging from undocked to docked
- Changing from undocked to split via menu
- Changing from undocked to floating via menu
- Pinched from undocked to floating
- Changing from undocked to expanded shortcuts bar via menu button
- Rotating orientation while keyboard open
Split Keyboard (iPad only)
- Changing from docked to split via menu
- Dragged from docked to split
- Opening/Closing
- Dragging to a new position
- Changing from split to docked via menu
- Dragging from split to docked
- Changing from split to undocked via menu
- Pinching from split to undocked
- Dragging from split to split docked
- Changing from split to floating via menu
- Changing from split to expanded shortcuts bar via menu button
- Rotating orientation while keyboard open
Floating Keyboard (iPad only)
- Changing from docked to floating via menu
- Pinching from docked to floating
- Opening/Closing
- Dragging floating to docked
- Spreading floating to docked
- Rotating orientation while keyboard open
Expanded Shortcuts Bar (iPad only)
- Changing from docked to expanded shortcuts bar via menu button
- Changing from minimized shortcuts bar to expanded shortcuts bar via menu button
- Opening/Closing
- Dragging from expanded shortcuts bar to minimized shortcuts bar
- Rotating orientation while keyboard open
Minimized Shortcuts Bar (iPad only)
- Changing from expanded shortcuts bar to minimized shortcuts bar via menu
- Opening/Closing
- Dragging to a new position
- Dragging from minimized shortcuts bar to expanded shortcuts bar
- Rotating orientation while keyboard open
One-handed (iPhone only)
- Changing from docked to one-handed right
- Changing from docked to one-handed left
- Opening/Closing
- Changing from one-handed right to docked
- Changing from one-handed left to docked
Switching between states
- Open original app, switch to second app, open keyboard, switch back to original app, open keyboard, switch to second app, switch to original app
- Open original app, switch to second app, open keyboard, switch back to original app, switch to second app, switch to original app
- Swipe down to open Control Center, close control center and click on input field quickly

Devices I tested with:

Simulator iPhone X (iOS 12.4)
Simulator iPhone 11 Pro (iOS 13.0)
Simulator iPhone 12 Pro (iOS 14.3)
Simulator iPhone 14 Pro (iOS 16.1)
Simulator iPad Pro 9.7in (iOS 12.4)
Simulator iPad Pro 9.7in (iOS 14.3)
Simulator iPad 9th gen (iOS 16.1)
Physical iPad Pro 11 in (iOS 16.1)

Sample code I used for the flutter app: flutter/flutter#104689 (comment)

Custom keyboard extension code sample

import UIKit

class KeyboardViewController: UIInputViewController {

    @IBOutlet var nextKeyboardButton: UIButton!
    @IBOutlet var closeKeyboardButton: UIButton!
    
    override func updateViewConstraints() {
        super.updateViewConstraints()

        let expandedHeight: CGFloat = 397 + (2/3)
        let constraint = NSLayoutConstraint(item: self.view, attribute: .height, relatedBy: .equal, toItem: nil, attribute: .notAnAttribute, multiplier: 0.0, constant: expandedHeight)
        self.view.addConstraint(constraint)
    }
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        self.nextKeyboardButton = UIButton(type: .system)
        self.nextKeyboardButton.setTitle(NSLocalizedString("Next Keyboard", comment: "Title for 'Next Keyboard' button"), for: [])
        self.nextKeyboardButton.sizeToFit()
        self.nextKeyboardButton.translatesAutoresizingMaskIntoConstraints = false
        self.nextKeyboardButton.addTarget(self, action: #selector(handleInputModeList(from:with:)), for: .allTouchEvents)
        self.view.addSubview(self.nextKeyboardButton)
        self.nextKeyboardButton.leftAnchor.constraint(equalTo: self.view.leftAnchor).isActive = true
        self.nextKeyboardButton.bottomAnchor.constraint(equalTo: self.view.bottomAnchor).isActive = true
        
        self.closeKeyboardButton = UIButton(type: .system)
        self.closeKeyboardButton.setTitle(NSLocalizedString("Close Keyboard", comment: "Title for 'Close Keyboard' button"), for: [])
        self.closeKeyboardButton.sizeToFit()
        self.closeKeyboardButton.translatesAutoresizingMaskIntoConstraints = false
        self.closeKeyboardButton.addTarget(self, action: #selector(closeKeyboard), for: .allTouchEvents)
        self.view.addSubview(self.closeKeyboardButton)
        self.closeKeyboardButton.rightAnchor.constraint(equalTo: self.view.rightAnchor).isActive = true
        self.closeKeyboardButton.bottomAnchor.constraint(equalTo: self.view.bottomAnchor).isActive = true
    }
    
    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
    }
    
    override func viewWillLayoutSubviews() {
        self.nextKeyboardButton.isHidden = !self.needsInputModeSwitchKey
        super.viewWillLayoutSubviews()
    }
    
    @objc func closeKeyboard() {
        self.textDocumentProxy.insertText("\n")
    }
}

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

jmagman · 2022-11-18T22:54:44Z

Adding @cyanglaz.
@luckysmg may have input.

luckysmg · 2022-11-19T04:25:39Z

Thx @vashworth for working on fixing this.

(Just some advices and sharing my thoughts, maybe not very appropriate😄).

I think now we will deal with different situtation in keyboard notification function, and we use some judgement to judge the keyboard is docked or undocked or floating. This is ok, emm but maybe not very clear, and it is not very easy to know current judgement branch will impact which scenario...

I think maybe we can do sth to make this more clear like this

In this scenario,we can define an class or enum, maybe named with: KeyboardAttachMode
And we compute a KeyboardAttachMode using the NSNotification passed by keyboard notification(where we can get the keyboard frame from)
And final we can calculate the end frame by using the keyboard attach mode and notificaition.

Here is the some demo code.

Enum KeyboardAttachMode {
   docked,
   undocked,
   floating
   .......
}

- (KeyboardAttachMode)computeKeyboardAttachMode:(NSNotification noti){
       CGRect  endFrame = noti["UIKeyboardEndFrameUserKey"]
       if (endFrame xxxxxx) {
           return KeyboardAttachMode.docked;
      } else if (xxxx) {
           return KeyboardAttachMode.floating;
      }
}

- (double)computeInset(KeyboardAttachMode mode, NSNotification notification) {
   double inset = 0;
   if (mode == .docked) {
        inset = xxxxxx
   } else mode == .floating{
   } else .....{
  }
  return inset;
}

- (void)keyboardShow {
    KeyboardAttachMode mode = computeKeyboardAttachMode(noti);
    double computeInset = computeInset(mode,noti);
     if (computeInset != self.targetButtomInset) {
        self.targetBottomInset = computeInset;
        startKeyboardAnimation();
     }
}

- (void)keyboardWillChangeFrame {
    KeyboardAttachMode mode = computeKeyboardAttachMode(noti);
    double computeInset = computeInset(mode,noti);
     if (computeInset != self.targetButtomInset) {
        self.targetBottomInset = computeInset;
        startKeyboardAnimation();
     }
}

- (void)keyboardWillHide {
    KeyboardAttachMode mode = computeKeyboardAttachMode(noti);
    double computeInset = computeInset(mode,noti);
     if (computeInset != self.targetButtomInset) {
        self.targetBottomInset = computeInset;
        startKeyboardAnimation();
     }
}

I think this will be more clear O(∩_∩)O

shell/platform/darwin/ios/framework/Source/FlutterViewControllerTest.mm

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

cyanglaz

Is it possible to use scenario app test to do some UITests? Maybe have a UITextField as PlatformView to open up keyboard, undock the keyboard and take a screenshot?

I think you mentioned this in the team meeting, is it not possible because there is no way to make the keyboard undock in XCUITests?

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

cyanglaz · 2022-11-20T18:41:25Z

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

+- (void)keyboardWillChangeFrame:(NSNotification*)notification {
+  // Immediately prior to a change in keyboard frame, this notification is triggered.
+  // There are some cases where UIKeyboardWillShowNotification & UIKeyboardWillHideNotification
+  // do not act as expected and this is used to catch those cases.


Does the change frame notification work in all cases? Maybe we should eliminate UIKeyboardWillShowNotification and UIKeyboardWillHideNotification if the change frame notification is more reliable?

Could be a nice refactor in a follow up PR if we can use frame change notification for all cases.

So none of them are very reliable which is why I used all three of them. Here's some of the use cases for example

When predictive-only/minimized keyboard is dragged, it sends a UIKeyboardWillShowNotification notification where the keyboard = CGRectZero, which we use to determine it's "floating" and should have inset 0 . Whereas in UIKeyboardWillChangeFrameNotification we skip when keyboard = CGRectZero, because otherwise it will set inset to 0 prematurely in some cases like when you dragged a docked keyboard but don't actually change it from docked.

When keyboard is changed from docked to floating, it's supposed to send a UIKeyboardWillHideNotification notification but it doesn't, so we handle in UIKeyboardWillChangeFrameNotification instead

When keyboard goes from docked to undocked, the UIKeyboardWillChangeFrameNotification position will not be completely below the screen so it will still have an inset, which is at least one reason why we also need UIKeyboardWillHideNotification

Ah, gotcha. I think logic this complex deserves a design documentation :) And we should have a comment in the code to link to the design doc. (The same way PlatformView should have and never did :p )

I think for this PR, it would be nice to explain "some cases" a little more in the comments. Or add a link in the code to your comment here: #37719 (comment)

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

shell/platform/darwin/ios/framework/Source/FlutterViewControllerTest.mm

cyanglaz · 2022-11-20T19:03:52Z

However, there are some cases where these events are missing, have incomplete values, or do something unexpected, so we use UIKeyboardWillShowNotification in addition to help handle those cases.

Did you mean "UIKeyboardWillChangeFrameNotification"? :)

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

shell/platform/darwin/ios/framework/Source/FlutterViewControllerTest.mm

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

…emove rotation logic

…if keyboard intersects with screen to accomodate for repeating decimals, format

jmagman · 2022-12-05T18:46:50Z

Failing tests:
	-[FlutterTextInputPluginTest testInputViewsHasNonNilInputDelegate]
	-[FlutterViewControllerTest testCalculateKeyboardInset]

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8795849902472179793/+/u/test:_Host_Tests_for_ios_debug_sim__3_/stdout

jmagman · 2022-12-07T18:46:27Z

Nice, tests now passing! Thanks for tracking those notification issues @vashworth.

jmagman

LGTM!

@luckysmg thanks so much for the thorough review on this one! Now that you have contributor permissions 🎉 can you mark when it's up to your approval standards?

@vashworth let's hold off merging this until that review is complete.

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

hellohuanlin

Left a few questions.

Your design doc is really helpful for me to understand the code. It makes the PR review a lot easier! Great work!

hellohuanlin · 2022-12-09T19:30:48Z

shell/platform/darwin/ios/framework/Source/FlutterPlatformPluginTest.mm

      [self expectationWithDescription:@"isLiveTextInputAvailableInvoke"];
  FlutterPlatformPlugin* plugin =
-      [[FlutterPlatformPlugin alloc] initWithEngine:_weakFactory->GetWeakPtr()];
+      [[[FlutterPlatformPlugin alloc] initWithEngine:_weakFactory->GetWeakPtr()] autorelease];


thanks for fixing all of these.

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

hellohuanlin · 2022-12-09T19:46:19Z

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

-
-  // Ignore keyboard notifications if engine’s viewController is not current viewController.
+  // Engine’s viewController is not current viewController.
  if ([_engine.get() viewController] != self) {


does this happen when an app shows multiple flutter view controller on the same screen?

Yeah, from my understanding this can happen when you're using add-to-app stuff like this flutter/flutter#39036 (comment). This is a good point, though. I didn't test use case like that.

Update: I did test this use case manually and everything appeared to be working correctly.

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

luckysmg · 2022-12-10T00:34:31Z

Mac mac_ios_engine_release failed, maybe need to rebase to main branch ?

luckysmg

LGTM O(∩_∩)O

cyanglaz

LGTM!

jmagman

Thank you for tracking all these corner cases down! LGTM!

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm

hellohuanlin

Thanks for making those changes. Your design doc is very helpful reviewing the PR.

… loaded, update a comment

vashworth added 2 commits November 17, 2022 10:30

fix keyboard inset not collapsing when expected

081a812

fix some formatting

8501ff2

flutter-dashboard bot added the platform-ios label Nov 17, 2022

vashworth added 4 commits November 17, 2022 14:37

fix issue with rotating with undocked and split keyboard

0c0cc73

fix formatting

078308c

fix behavior on slide over view

6f16d3d

fix formatting

79705b4

vashworth marked this pull request as ready for review November 18, 2022 20:17

vashworth requested a review from jmagman November 18, 2022 20:20

jmagman reviewed Nov 18, 2022

View reviewed changes

jmagman requested a review from cyanglaz November 18, 2022 22:54

luckysmg reviewed Nov 20, 2022

View reviewed changes

shell/platform/darwin/ios/framework/Source/FlutterViewControllerTest.mm Outdated Show resolved Hide resolved

luckysmg reviewed Nov 20, 2022

View reviewed changes

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm Outdated Show resolved Hide resolved

cyanglaz reviewed Nov 20, 2022

View reviewed changes

hellohuanlin mentioned this pull request Nov 22, 2022

The keyboard of other apps affected the flutter app flutter/flutter#115528

Closed

refactor to make logic more clear

6270428

jmagman reviewed Nov 28, 2022

View reviewed changes

luckysmg reviewed Nov 29, 2022

View reviewed changes

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm Outdated Show resolved Hide resolved

cyanglaz reviewed Nov 30, 2022

View reviewed changes

vashworth added 3 commits December 2, 2022 12:23

move enum to header file, remove unneeded parameters, syntax fixes, r…

084b228

…emove rotation logic

ignore notification if app state is not active, change way it checks …

cbece73

…if keyboard intersects with screen to accomodate for repeating decimals, format

Merge remote-tracking branch 'upstream/main' into keyboard_fixes

6c4f65e

vashworth added 2 commits December 6, 2022 17:45

fix leaking unit test

39313a7

use viewIfLoaded and update tests to fix mocking

602c703

jmagman approved these changes Dec 7, 2022

View reviewed changes

luckysmg reviewed Dec 8, 2022

View reviewed changes

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm Outdated Show resolved Hide resolved

hellohuanlin reviewed Dec 9, 2022

View reviewed changes

change ignore logic related to application state to be more specific

ea788a1

luckysmg reviewed Dec 10, 2022

View reviewed changes

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm Outdated Show resolved Hide resolved

vashworth added 2 commits December 12, 2022 10:09

Merge remote-tracking branch 'upstream/main' into keyboard_fixes

179f510

add more comments

5aa9221

luckysmg approved these changes Dec 12, 2022

View reviewed changes

add more comments

010e86b

cyanglaz approved these changes Dec 13, 2022

View reviewed changes

jmagman approved these changes Dec 13, 2022

View reviewed changes

hellohuanlin reviewed Dec 13, 2022

View reviewed changes

shell/platform/darwin/ios/framework/Source/FlutterViewController.mm Outdated Show resolved Hide resolved

hellohuanlin approved these changes Dec 13, 2022

View reviewed changes

change function name to be more clear, add warning log if view is not…

aa6276e

… loaded, update a comment

vashworth added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 14, 2022

auto-submit bot merged commit 479bb73 into flutter:main Dec 14, 2022

engine-flutter-autoroll mentioned this pull request Dec 14, 2022

Roll Flutter Engine from 0a6a4a58f4f7 to db5605ea7115 (11 revisions) flutter/flutter#117109

Merged

luckysmg mentioned this pull request Dec 15, 2022

iOS keyboard animation synchronization #37604

Merged

8 tasks

jmagman mentioned this pull request Dec 15, 2022

[iOS] [iPad] fix avoid bottom inset in split view mode #35535

Closed

8 tasks

vashworth mentioned this pull request Dec 19, 2022

[iPadOS] Stage Manager has incorrect inset when in top corner flutter/flutter#117324

Open

vashworth mentioned this pull request Feb 22, 2023

Implement a way for getting keyboard final display height on mobile platform. flutter/flutter#121087

Open

huycozy mentioned this pull request Sep 18, 2023

[iOS] Keyboard partially overlaps TextField after native dialog closes flutter/flutter#133537

Open

2 tasks

vashworth mentioned this pull request Mar 21, 2025

Investigate adopting iOS Keyboard Layout Guide flutter/flutter#165672

Open

Fix issues related to keyboard inset #37719

Fix issues related to keyboard inset #37719

Uh oh!

Conversation

vashworth commented Nov 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmagman commented Nov 18, 2022

Uh oh!

luckysmg commented Nov 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cyanglaz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cyanglaz Nov 20, 2022

Choose a reason for hiding this comment

Uh oh!

vashworth Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

cyanglaz Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cyanglaz commented Nov 20, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmagman commented Dec 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmagman commented Dec 7, 2022

Uh oh!

jmagman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hellohuanlin left a comment

Choose a reason for hiding this comment

Uh oh!

hellohuanlin Dec 9, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hellohuanlin Dec 9, 2022

vashworth commented Nov 17, 2022 •

edited

Loading

luckysmg commented Nov 19, 2022 •

edited

Loading

jmagman commented Dec 5, 2022 •

edited

Loading